[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:46 dist-test-slave-rc9f krb5kdc[5202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:46 dist-test-slave-rc9f krb5kdc[5202](info): set up 2 sockets
Sep 28 21:16:46 dist-test-slave-rc9f krb5kdc[5202](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:16:47.122354 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:16:47.234568 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:32813
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:32813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:47.353047 5215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:47.353323 5215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:47.353363 5215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:47.357326 5215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:47.357384 5215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:47.357402 5215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:47.357419 5215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:47.357432 5215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:47.362664 5215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:32813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:32813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:47.363644 5215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:47.364594 5215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:47.372845 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.372869 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.372938 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:47.373045 5215 server_base.cc:1034] running on GCE node
I20260928 21:16:47.373669 5215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:47.374679 5215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:47.375828 5215 hybrid_clock.cc:648] HybridClock initialized: now 1790630207375811 us; error 29 us; skew 500 ppm
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:47.378458 5215 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:16:47.379297 5215 webserver.cc:460] Webserver started at http://127.5.18.254:34155/ using document root <none> and password file <none>
I20260928 21:16:47.379834 5215 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:47.379894 5215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:47.380089 5215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:47.382079 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "8d08a569cfef466a9d04f8f6dbb8de89"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "c599f65df1a5137cea8b8b8c07517ead"
server_key_iv: "83a002163c5edb9d697b15d3697f1069"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.382611 5215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "8d08a569cfef466a9d04f8f6dbb8de89"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "c599f65df1a5137cea8b8b8c07517ead"
server_key_iv: "83a002163c5edb9d697b15d3697f1069"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.386111 5215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:47.388293 5230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.389631 5215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:47.389732 5215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "8d08a569cfef466a9d04f8f6dbb8de89"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "c599f65df1a5137cea8b8b8c07517ead"
server_key_iv: "83a002163c5edb9d697b15d3697f1069"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.389828 5215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:47.403270 5215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:47.406322 5215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:47.406497 5215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:47.413086 5215 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:32813
I20260928 21:16:47.413103 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:32813 every 8 connection(s)
I20260928 21:16:47.414116 5215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:47.416874 5283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:47.420375 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5215
I20260928 21:16:47.420506 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:16:47.421793 5195 external_mini_cluster.cc:1428] Setting key efb3dc77db8f3956c0a1a1a62d7b5487
I20260928 21:16:47.422402 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89: Bootstrap starting.
I20260928 21:16:47.425086 5283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:47.426000 5283 log.cc:824] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:47.427732 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89: No bootstrap required, opened a new log
I20260928 21:16:47.431492 5283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } }
I20260928 21:16:47.431717 5283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:47.431775 5283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d08a569cfef466a9d04f8f6dbb8de89, State: Initialized, Role: FOLLOWER
I20260928 21:16:47.432217 5283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } }
I20260928 21:16:47.432353 5283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:47.432415 5283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:47.432504 5283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:47.433355 5283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } }
I20260928 21:16:47.433697 5283 leader_election.cc:302] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d08a569cfef466a9d04f8f6dbb8de89; no voters:
I20260928 21:16:47.433975 5283 leader_election.cc:288] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:47.434060 5288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:47.434242 5288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [term 1 LEADER]: Becoming Leader. State: Replica: 8d08a569cfef466a9d04f8f6dbb8de89, State: Running, Role: LEADER
I20260928 21:16:47.434525 5288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } }
I20260928 21:16:47.435228 5283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:47.436170 5289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d08a569cfef466a9d04f8f6dbb8de89. Latest consensus state: current_term: 1 leader_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } } }
I20260928 21:16:47.436290 5289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:47.436532 5290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d08a569cfef466a9d04f8f6dbb8de89" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 32813 } } }
I20260928 21:16:47.436622 5290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:47.436715 5297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:47.439996 5297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:47.446213 5297 catalog_manager.cc:1347] Generated new cluster ID: c8393865310c4765a3eaf620da38641a
I20260928 21:16:47.446280 5297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:47.448161 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:47.424559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.5.18.254:32813)
0928 21:16:47.424958 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:47.424973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:47.426268 (+ 1295us) server_negotiation.cc:407] Connection header received
0928 21:16:47.431058 (+ 4790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:47.431079 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:47.431456 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:47.431779 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:47.434228 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.435307 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.436987 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.437696 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.437850 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:47.442659 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:47.442664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:47.442675 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:47.442701 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:47.444419 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.444925 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.444930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.444934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.445055 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.445404 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.445406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.445407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.445862 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:47.446065 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:47.446966 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 21:16:47.447265 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260928 21:16:47.466593 5297 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:47.467487 5297 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:47.472287 5297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d08a569cfef466a9d04f8f6dbb8de89: Generated new TSK 0
I20260928 21:16:47.473073 5297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:16:47.505813 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:47.613238 5309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:47.613478 5309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:47.613524 5309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:47.617430 5309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:47.617480 5309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:47.617549 5309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:16:47.622627 5309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:47.623564 5309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:47.624526 5309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:47.633662 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.633706 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.633767 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:47.633827 5309 server_base.cc:1034] running on GCE node
I20260928 21:16:47.634289 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:47.634864 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:47.636023 5309 hybrid_clock.cc:648] HybridClock initialized: now 1790630207636000 us; error 34 us; skew 500 ppm
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:47.638621 5309 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:16:47.639467 5309 webserver.cc:460] Webserver started at http://127.5.18.193:34095/ using document root <none> and password file <none>
I20260928 21:16:47.639979 5309 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:47.640043 5309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:47.640223 5309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:47.642225 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "59ef39c97cc74668ac11de4f8392893c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "5895a90933345fec1ec7353954526e30"
server_key_iv: "2b94cb4c53c2e2e5aef16f71a606f2eb"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.642750 5309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "59ef39c97cc74668ac11de4f8392893c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "5895a90933345fec1ec7353954526e30"
server_key_iv: "2b94cb4c53c2e2e5aef16f71a606f2eb"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.646258 5309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:47.648627 5324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.649859 5309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 21:16:47.649958 5309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "59ef39c97cc74668ac11de4f8392893c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "5895a90933345fec1ec7353954526e30"
server_key_iv: "2b94cb4c53c2e2e5aef16f71a606f2eb"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.650050 5309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:47.667914 5309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:47.670533 5309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:47.670701 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:47.671245 5309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:47.672034 5309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:47.672078 5309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.672129 5309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:47.672154 5309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.680367 5309 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:39669
I20260928 21:16:47.680388 5437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:39669 every 8 connection(s)
I20260928 21:16:47.681197 5309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:47.691274 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5309
I20260928 21:16:47.691380 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:47.691612 5195 external_mini_cluster.cc:1428] Setting key 72bf8323191e75c634ed1f137e78441a
I20260928 21:16:47.694115 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:47.682887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:39731 (local address 127.5.18.254:32813)
0928 21:16:47.683000 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:47.683004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:47.683812 (+ 808us) server_negotiation.cc:407] Connection header received
0928 21:16:47.685131 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:47.685134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:47.685195 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:47.685261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:47.686449 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.686967 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.688063 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.688283 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.688376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:47.690578 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:47.690581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:47.690583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:47.690607 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:47.691880 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.692382 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.692385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.692387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.692458 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.692955 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.692959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.692960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.693178 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:47.693283 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:47.693846 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 21:16:47.693964 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:16:47.695031 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:47.683226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:32813 (local address 127.5.18.193:39731)
0928 21:16:47.683674 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:47.683715 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:47.684929 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:47.685438 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:47.685445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:47.685842 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:47.686306 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:47.686323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.687079 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.687082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:47.687949 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:47.687958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.688422 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.688426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:47.688575 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:47.689064 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:47.689079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:47.690430 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:47.691982 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:47.691988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:47.692002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:47.692290 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:47.692536 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:47.692539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:47.692541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:47.692681 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:47.693309 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:47.693315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:47.693544 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 21:16:47.693782 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 21:16:47.694093 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260928 21:16:47.696422 5438 heartbeater.cc:347] Connected to a master server at 127.5.18.254:32813
I20260928 21:16:47.696643 5438 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:47.697108 5438 heartbeater.cc:511] Master 127.5.18.254:32813 requested a full tablet report, sending...
I20260928 21:16:47.698535 5247 ts_manager.cc:194] Registered new tserver with Master: 59ef39c97cc74668ac11de4f8392893c (127.5.18.193:39669)
I20260928 21:16:47.699620 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:39731
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:16:47.714442 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:47.817698 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:47.817934 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:47.817976 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:47.821889 5443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:47.821938 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:47.822006 5443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:16:47.826956 5443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:47.827877 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:47.828800 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:47.837419 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.837445 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:47.837400 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:47.837723 5443 server_base.cc:1034] running on GCE node
I20260928 21:16:47.838084 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:47.838644 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:47.839787 5443 hybrid_clock.cc:648] HybridClock initialized: now 1790630207839769 us; error 29 us; skew 500 ppm
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:47.842176 5443 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:16:47.842942 5443 webserver.cc:460] Webserver started at http://127.5.18.194:34413/ using document root <none> and password file <none>
I20260928 21:16:47.843416 5443 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:47.843469 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:47.843623 5443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:47.845439 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "d9334292198a4cb6baaec537d894f20c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "03cb271f9dff4992ed78a72defde1790"
server_key_iv: "d0386c3c45faa0641ce679e4f3f0133d"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.845978 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "d9334292198a4cb6baaec537d894f20c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "03cb271f9dff4992ed78a72defde1790"
server_key_iv: "d0386c3c45faa0641ce679e4f3f0133d"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.849262 5443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:47.851455 5458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.852577 5443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:47.852679 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "d9334292198a4cb6baaec537d894f20c"
format_stamp: "Formatted at 2026-09-28 21:16:47 on dist-test-slave-rc9f"
server_key: "03cb271f9dff4992ed78a72defde1790"
server_key_iv: "d0386c3c45faa0641ce679e4f3f0133d"
server_key_version: "encryptionkey@0"
I20260928 21:16:47.852770 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:47.899564 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:47.902437 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:47.902603 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:47.903183 5443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:47.903965 5443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:47.904011 5443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.904052 5443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:47.904080 5443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:47.912185 5443 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:35449
I20260928 21:16:47.912213 5571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:35449 every 8 connection(s)
I20260928 21:16:47.913014 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:47.919653 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5443
I20260928 21:16:47.919761 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:47.920042 5195 external_mini_cluster.cc:1428] Setting key 29e10d35b7d563b8c7528d07c5f43dba
Sep 28 21:16:47 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:47.925731 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:47.914618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:46011 (local address 127.5.18.254:32813)
0928 21:16:47.914750 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:47.914755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:47.915586 (+ 831us) server_negotiation.cc:407] Connection header received
0928 21:16:47.916812 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:47.916814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:47.916858 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:47.916925 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:47.918069 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.918582 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.919636 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.919941 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.920042 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:47.922468 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:47.922471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:47.922473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:47.922496 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:47.923877 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.924324 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.924328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.924329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.924404 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:47.924694 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:47.924697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:47.924699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:47.924886 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:47.924971 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:47.925428 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 21:16:47.925545 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:47.926473 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:47.915032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:32813 (local address 127.5.18.194:46011)
0928 21:16:47.915445 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:47.915485 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:47.916632 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:47.917072 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:47.917078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:47.917428 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:47.917938 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:47.917952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.918668 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.918671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:47.919533 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:47.919541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:47.920067 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:47.920070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:47.920231 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:47.920757 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:47.920773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:47.922300 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:47.923973 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:47.923977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:47.923988 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:47.924225 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:47.924484 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:47.924487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:47.924489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:47.924611 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:47.924981 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:47.924988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:47.925202 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 21:16:47.925373 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 21:16:47.925697 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260928 21:16:47.927496 5572 heartbeater.cc:347] Connected to a master server at 127.5.18.254:32813
I20260928 21:16:47.927697 5572 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:47.928138 5572 heartbeater.cc:511] Master 127.5.18.254:32813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:16:47.929049 5247 ts_manager.cc:194] Registered new tserver with Master: d9334292198a4cb6baaec537d894f20c (127.5.18.194:35449)
I20260928 21:16:47.929651 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:46011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:16:47.944675 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:48.049008 5577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:48.049229 5577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:48.049269 5577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:48.053231 5577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:48.053277 5577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:48.053344 5577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:16:48.058444 5577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:32813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:48.059365 5577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:48.060243 5577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:48.068892 5585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:48.069018 5582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:48.068892 5583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:48.069134 5577 server_base.cc:1034] running on GCE node
I20260928 21:16:48.069500 5577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:48.070070 5577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:48.071230 5577 hybrid_clock.cc:648] HybridClock initialized: now 1790630208071209 us; error 37 us; skew 500 ppm
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:48.073717 5577 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:16:48.074484 5577 webserver.cc:460] Webserver started at http://127.5.18.195:40961/ using document root <none> and password file <none>
I20260928 21:16:48.074970 5577 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:48.075027 5577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:48.075215 5577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:48.077124 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "17a3eb6d8a85495b8e19867d413d5f10"
format_stamp: "Formatted at 2026-09-28 21:16:48 on dist-test-slave-rc9f"
server_key: "0acfe0e4ba1ef10c3348c202777eafd9"
server_key_iv: "026bcc31aa23d60386b64b32460e8c64"
server_key_version: "encryptionkey@0"
I20260928 21:16:48.077674 5577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "17a3eb6d8a85495b8e19867d413d5f10"
format_stamp: "Formatted at 2026-09-28 21:16:48 on dist-test-slave-rc9f"
server_key: "0acfe0e4ba1ef10c3348c202777eafd9"
server_key_iv: "026bcc31aa23d60386b64b32460e8c64"
server_key_version: "encryptionkey@0"
I20260928 21:16:48.081099 5577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:48.083334 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:48.084436 5577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:48.084530 5577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "17a3eb6d8a85495b8e19867d413d5f10"
format_stamp: "Formatted at 2026-09-28 21:16:48 on dist-test-slave-rc9f"
server_key: "0acfe0e4ba1ef10c3348c202777eafd9"
server_key_iv: "026bcc31aa23d60386b64b32460e8c64"
server_key_version: "encryptionkey@0"
I20260928 21:16:48.084621 5577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:48.115252 5577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:48.118110 5577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:48.118316 5577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:48.118914 5577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:48.119794 5577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:48.119846 5577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:48.119905 5577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:48.119933 5577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:48.128928 5577 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:40171
I20260928 21:16:48.128945 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:40171 every 8 connection(s)
I20260928 21:16:48.129915 5577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:48.140025 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5577
I20260928 21:16:48.140136 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:48.140376 5195 external_mini_cluster.cc:1428] Setting key 20e5cace9034db261962e8285d5485f3
I20260928 21:16:48.142988 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:48.131618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:48315 (local address 127.5.18.254:32813)
0928 21:16:48.131733 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:48.131736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:48.132645 (+ 909us) server_negotiation.cc:407] Connection header received
0928 21:16:48.133946 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:48.133949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:48.134008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:48.134075 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:48.135211 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.135777 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.136930 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.137139 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.137222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:48.139493 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:48.139496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:48.139498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:48.139523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:48.140945 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.141492 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.141495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.141497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.141605 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.141923 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.141927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.141929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.142113 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:48.142190 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:48.142718 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 21:16:48.142839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:16:48.143724 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:48.131998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:32813 (local address 127.5.18.195:48315)
0928 21:16:48.132437 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:48.132492 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:48.133741 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:48.134251 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:48.134258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:48.134613 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:48.135072 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:48.135085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.135894 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.135897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:48.136822 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:48.136832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.137264 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.137268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:48.137435 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:48.137978 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:48.137994 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:48.139337 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:48.141055 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:48.141068 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:48.141079 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:48.141404 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:48.141688 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:48.141691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:48.141692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:48.141844 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:48.142204 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:48.142210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:48.142494 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 21:16:48.142663 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 21:16:48.142969 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260928 21:16:48.144656 5706 heartbeater.cc:347] Connected to a master server at 127.5.18.254:32813
I20260928 21:16:48.144852 5706 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:48.145263 5706 heartbeater.cc:511] Master 127.5.18.254:32813 requested a full tablet report, sending...
I20260928 21:16:48.146183 5247 ts_manager.cc:194] Registered new tserver with Master: 17a3eb6d8a85495b8e19867d413d5f10 (127.5.18.195:40171)
I20260928 21:16:48.146708 5247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:48315
I20260928 21:16:48.154424 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:48.177656 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:48.169107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.5.18.193:39669)
0928 21:16:48.169385 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:48.169399 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:48.169424 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:16:48.169582 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:48.169590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:48.169787 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:48.169953 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:48.170560 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.171315 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.172111 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.172472 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.172563 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:48.174700 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:48.174708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:48.174719 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:48.174742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:48.176044 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.176422 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.176427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.176429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.176504 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.176730 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.176734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.176735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.176963 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:48.177163 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:48.177321 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:16:48.177464 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
W20260928 21:16:48.178558 5352 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35768
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:48.201207 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:48.193852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.5.18.193:39669)
0928 21:16:48.193979 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:48.193983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:48.194096 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:16:48.194227 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:48.194230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:48.194272 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:48.194349 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:48.194991 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.195503 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.196303 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:48.196528 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:48.196611 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:48.198722 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:48.198725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:48.198727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:48.198748 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:48.199815 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.200165 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.200169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.200170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.200239 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:48.200471 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:48.200474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:48.200476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:48.200627 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:48.200739 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:48.200954 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:16:48.201061 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:48.202047 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5309
I20260928 21:16:48.206552 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5443
I20260928 21:16:48.210497 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5577
I20260928 21:16:48.214596 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5215
2026-09-28T21:16:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2056 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5720](info): set up 2 sockets
Sep 28 21:16:48 dist-test-slave-rc9f krb5kdc[5720](info): commencing operation
krb5kdc: starting...
W20260928 21:16:49.152894 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:16:49.262638 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42825
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:49.368577 5733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:49.368824 5733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:49.368861 5733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:49.372773 5733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:49.372822 5733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:49.372840 5733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:49.372857 5733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:49.372884 5733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:49.377943 5733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:49.378906 5733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:49.379855 5733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:49.387900 5738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:49.387967 5733 server_base.cc:1034] running on GCE node
W20260928 21:16:49.387997 5739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:49.387912 5741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:49.388631 5733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:49.389693 5733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:49.390832 5733 hybrid_clock.cc:648] HybridClock initialized: now 1790630209390811 us; error 37 us; skew 500 ppm
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:49.393252 5733 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:16:49.394140 5733 webserver.cc:460] Webserver started at http://127.5.18.254:44271/ using document root <none> and password file <none>
I20260928 21:16:49.394613 5733 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:49.394665 5733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:49.394832 5733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:49.396665 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "2838919f118848dfab15a860124e3bbb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "28c24fa3ac03984e2de70a0156d9abd6"
server_key_iv: "e19504f69ba0bc5d3a1c996d95cb26bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.397177 5733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "2838919f118848dfab15a860124e3bbb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "28c24fa3ac03984e2de70a0156d9abd6"
server_key_iv: "e19504f69ba0bc5d3a1c996d95cb26bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.400465 5733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 21:16:49.402585 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.403674 5733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:49.403769 5733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "2838919f118848dfab15a860124e3bbb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "28c24fa3ac03984e2de70a0156d9abd6"
server_key_iv: "e19504f69ba0bc5d3a1c996d95cb26bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.403864 5733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:49.429391 5733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:49.431954 5733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:49.432118 5733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:49.438696 5733 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:42825
I20260928 21:16:49.438714 5800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:42825 every 8 connection(s)
I20260928 21:16:49.439580 5733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:49.442224 5801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:49.447492 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: Bootstrap starting.
I20260928 21:16:49.447860 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5733
I20260928 21:16:49.447980 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:16:49.448199 5195 external_mini_cluster.cc:1428] Setting key 02e865898629b26407cd202b7cf381fc
I20260928 21:16:49.449935 5801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:49.450628 5801 log.cc:824] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:49.452595 5801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: No bootstrap required, opened a new log
I20260928 21:16:49.456295 5801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } }
I20260928 21:16:49.456503 5801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:49.456557 5801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2838919f118848dfab15a860124e3bbb, State: Initialized, Role: FOLLOWER
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:49.457031 5801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } }
I20260928 21:16:49.457193 5801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:49.457257 5801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:49.457357 5801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:49.458199 5801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } }
I20260928 21:16:49.458515 5801 leader_election.cc:302] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2838919f118848dfab15a860124e3bbb; no voters:
I20260928 21:16:49.458781 5801 leader_election.cc:288] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:49.458902 5806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:49.459095 5806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [term 1 LEADER]: Becoming Leader. State: Replica: 2838919f118848dfab15a860124e3bbb, State: Running, Role: LEADER
I20260928 21:16:49.459473 5806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } }
I20260928 21:16:49.459908 5801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:49.460583 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2838919f118848dfab15a860124e3bbb. Latest consensus state: current_term: 1 leader_uuid: "2838919f118848dfab15a860124e3bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } } }
I20260928 21:16:49.460709 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:49.461031 5807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2838919f118848dfab15a860124e3bbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2838919f118848dfab15a860124e3bbb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42825 } } }
I20260928 21:16:49.461122 5807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:49.462692 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:49.449453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.5.18.254:42825)
0928 21:16:49.449859 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:49.449874 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:49.449907 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:16:49.450531 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:49.450547 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:49.450895 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:49.451159 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:49.451828 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.453235 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.454069 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.454757 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.454926 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:49.457197 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:49.457204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:49.457215 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:49.457243 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:49.459049 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.459567 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.459573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.459579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.459709 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.460047 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.460050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.460052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.460545 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:49.460737 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:49.461467 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 21:16:49.461812 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
W20260928 21:16:49.464651 5821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:49.464735 5821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:49.464807 5814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:49.465704 5814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:49.470598 5814 catalog_manager.cc:1347] Generated new cluster ID: c45317f8e692441abd1e1b86a3d186d3
I20260928 21:16:49.470659 5814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:49.514129 5814 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:49.515100 5814 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:49.521303 5814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2838919f118848dfab15a860124e3bbb: Generated new TSK 0
I20260928 21:16:49.522130 5814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:16:49.560184 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:49.665195 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:49.665457 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:49.665504 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:49.669430 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:49.669483 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:49.669552 5827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:16:49.674628 5827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:49.675603 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:49.676615 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:49.687378 5827 server_base.cc:1034] running on GCE node
W20260928 21:16:49.687333 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:49.687328 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:49.687331 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:49.688072 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:49.688781 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:49.689945 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790630209689923 us; error 35 us; skew 500 ppm
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:49.692754 5827 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:16:49.693696 5827 webserver.cc:460] Webserver started at http://127.5.18.193:41207/ using document root <none> and password file <none>
I20260928 21:16:49.694227 5827 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:49.694285 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:49.694463 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:49.696477 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "0bce05fc2ee64a49ae1838cf25a84da2"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "e26ee0296b1485c1ad21176987d32fb7"
server_key_iv: "0455b09a47c9bbf835b455d8345612f7"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.696992 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "0bce05fc2ee64a49ae1838cf25a84da2"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "e26ee0296b1485c1ad21176987d32fb7"
server_key_iv: "0455b09a47c9bbf835b455d8345612f7"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.700630 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:16:49.702915 5842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.704061 5827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:49.704162 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "0bce05fc2ee64a49ae1838cf25a84da2"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "e26ee0296b1485c1ad21176987d32fb7"
server_key_iv: "0455b09a47c9bbf835b455d8345612f7"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.704249 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:49.723094 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:49.726281 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:49.726473 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:49.727043 5827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:49.727865 5827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:49.727911 5827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.727964 5827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:49.727994 5827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.736013 5827 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:44995
I20260928 21:16:49.736048 5955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:44995 every 8 connection(s)
I20260928 21:16:49.736917 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:49.745841 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5827
I20260928 21:16:49.745950 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:49.746228 5195 external_mini_cluster.cc:1428] Setting key c844ca03413eafeb870b3d43adf9059d
I20260928 21:16:49.750362 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:49.738667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:50737 (local address 127.5.18.254:42825)
0928 21:16:49.738806 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:49.738810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:49.740030 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 21:16:49.740846 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:49.740849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:49.740893 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:49.740964 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:49.742090 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.742610 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.743789 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.744041 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.744134 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:49.746695 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:49.746699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:49.746702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:49.746732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:49.748202 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.748773 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.748775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.748777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.748857 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.749241 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.749245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.749247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.749475 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:49.749596 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:49.750104 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 21:16:49.750230 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:16:49.751358 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:49.739041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42825 (local address 127.5.18.193:50737)
0928 21:16:49.739881 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:49.739923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:49.740669 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:49.741099 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:49.741106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:49.741450 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:49.741982 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:49.741997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.742742 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.742746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:49.743642 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:49.743652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.744211 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.744214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:49.744379 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:49.744937 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:49.744954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:49.746481 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:49.748325 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:49.748331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:49.748349 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:49.748687 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:49.748929 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:49.748932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:49.748934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:49.749132 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:49.749605 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:49.749612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:49.749875 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 21:16:49.750036 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 21:16:49.750331 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":507,"threads_started":1}
I20260928 21:16:49.752571 5956 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42825
I20260928 21:16:49.752795 5956 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:49.753258 5956 heartbeater.cc:511] Master 127.5.18.254:42825 requested a full tablet report, sending...
I20260928 21:16:49.754745 5765 ts_manager.cc:194] Registered new tserver with Master: 0bce05fc2ee64a49ae1838cf25a84da2 (127.5.18.193:44995)
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:16:49.755882 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:50737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:16:49.772255 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:49.877372 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:49.877624 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:49.877668 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:49.881482 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:49.881529 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:49.881615 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:16:49.886571 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:49.887482 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:49.888391 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:49.897439 5961 server_base.cc:1034] running on GCE node
W20260928 21:16:49.897393 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:49.897408 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:49.897393 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:49.897977 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:49.898571 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:49.899727 5961 hybrid_clock.cc:648] HybridClock initialized: now 1790630209899700 us; error 43 us; skew 500 ppm
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:49.902300 5961 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:16:49.903126 5961 webserver.cc:460] Webserver started at http://127.5.18.194:34953/ using document root <none> and password file <none>
I20260928 21:16:49.903644 5961 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:49.903705 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:49.903882 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:49.905857 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "57f537c208d84bf9af08a73663f783cb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "35a511a9b49b3adb1782ea7d7a57b980"
server_key_iv: "455a712742c3e5738f42e373a653357a"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.906378 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "57f537c208d84bf9af08a73663f783cb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "35a511a9b49b3adb1782ea7d7a57b980"
server_key_iv: "455a712742c3e5738f42e373a653357a"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.910074 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:49.912259 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.913375 5961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:49.913479 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "57f537c208d84bf9af08a73663f783cb"
format_stamp: "Formatted at 2026-09-28 21:16:49 on dist-test-slave-rc9f"
server_key: "35a511a9b49b3adb1782ea7d7a57b980"
server_key_iv: "455a712742c3e5738f42e373a653357a"
server_key_version: "encryptionkey@0"
I20260928 21:16:49.913582 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:49.923372 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:49.926235 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:49.926404 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:49.926900 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:49.927634 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:49.927677 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.927726 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:49.927754 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:49.935349 5961 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43105
I20260928 21:16:49.935384 6089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43105 every 8 connection(s)
I20260928 21:16:49.936152 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:49.937292 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 5961
I20260928 21:16:49.937397 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:49.937675 5195 external_mini_cluster.cc:1428] Setting key 1f8f3b839eb110f13da8c057507d93aa
Sep 28 21:16:49 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:16:49.949186 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:49.937953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45987 (local address 127.5.18.254:42825)
0928 21:16:49.938086 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:49.938091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:49.938972 (+ 881us) server_negotiation.cc:407] Connection header received
0928 21:16:49.940271 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:49.940274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:49.940327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:49.940395 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:49.941533 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.942084 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.943163 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.943390 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.943473 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:49.945936 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:49.945939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:49.945941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:49.945963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:49.947297 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.947721 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.947723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.947724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.947804 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:49.948141 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:49.948144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:49.948145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:49.948332 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:49.948417 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:49.948911 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 21:16:49.949026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:16:49.950032 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:49.938400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42825 (local address 127.5.18.194:45987)
0928 21:16:49.938837 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:49.938885 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:49.940072 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:49.940576 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:49.940583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:49.940911 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:49.941366 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:49.941382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.942198 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.942201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:49.943053 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:49.943061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:49.943516 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:49.943520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:49.943678 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:49.944184 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:49.944199 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:49.945787 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:49.947406 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:49.947410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:49.947421 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:49.947649 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:49.947887 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:49.947890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:49.947891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:49.948063 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:49.948434 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:49.948441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:49.948675 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 21:16:49.948852 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 21:16:49.949161 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260928 21:16:49.951093 6090 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42825
I20260928 21:16:49.951309 6090 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:49.951747 6090 heartbeater.cc:511] Master 127.5.18.254:42825 requested a full tablet report, sending...
I20260928 21:16:49.952733 5765 ts_manager.cc:194] Registered new tserver with Master: 57f537c208d84bf9af08a73663f783cb (127.5.18.194:43105)
I20260928 21:16:49.953270 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:45987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:16:49.961512 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:50.066084 6095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:50.066326 6095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:50.066368 6095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:50.070222 6095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:50.070271 6095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:50.070333 6095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:16:50.075341 6095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:50.076323 6095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:50.077244 6095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:50.086246 6103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:50.086325 6101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:50.086290 6095 server_base.cc:1034] running on GCE node
W20260928 21:16:50.086246 6100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:50.086812 6095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:50.087342 6095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:50.088492 6095 hybrid_clock.cc:648] HybridClock initialized: now 1790630210088473 us; error 28 us; skew 500 ppm
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:50.091038 6095 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:16:50.091828 6095 webserver.cc:460] Webserver started at http://127.5.18.195:42947/ using document root <none> and password file <none>
I20260928 21:16:50.092324 6095 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:50.092381 6095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:50.092538 6095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:50.094590 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "18b682a6f1334135989080d4c2677b11"
format_stamp: "Formatted at 2026-09-28 21:16:50 on dist-test-slave-rc9f"
server_key: "2fad1de30fa163da6ccc0c64896c7077"
server_key_iv: "d67b50f53cae4808ff763b4963df6fd4"
server_key_version: "encryptionkey@0"
I20260928 21:16:50.095089 6095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "18b682a6f1334135989080d4c2677b11"
format_stamp: "Formatted at 2026-09-28 21:16:50 on dist-test-slave-rc9f"
server_key: "2fad1de30fa163da6ccc0c64896c7077"
server_key_iv: "d67b50f53cae4808ff763b4963df6fd4"
server_key_version: "encryptionkey@0"
I20260928 21:16:50.098505 6095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:50.100654 6110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:50.101792 6095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:50.101893 6095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "18b682a6f1334135989080d4c2677b11"
format_stamp: "Formatted at 2026-09-28 21:16:50 on dist-test-slave-rc9f"
server_key: "2fad1de30fa163da6ccc0c64896c7077"
server_key_iv: "d67b50f53cae4808ff763b4963df6fd4"
server_key_version: "encryptionkey@0"
I20260928 21:16:50.101982 6095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:50.124882 6095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:50.127614 6095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:50.127774 6095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:50.128315 6095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:50.129086 6095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:50.129135 6095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:50.129187 6095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:50.129217 6095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:50.137106 6095 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:33901
I20260928 21:16:50.137151 6223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:33901 every 8 connection(s)
I20260928 21:16:50.137995 6095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:50.146957 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6095
I20260928 21:16:50.147053 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:50.147292 5195 external_mini_cluster.cc:1428] Setting key 058737c9258b49f046e6264ea3465a5d
I20260928 21:16:50.150516 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.139638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:49827 (local address 127.5.18.254:42825)
0928 21:16:50.139753 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.139756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.140517 (+ 761us) server_negotiation.cc:407] Connection header received
0928 21:16:50.141806 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.141810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.141867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.141935 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:50.143048 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.143563 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.144647 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.144859 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.144945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.147187 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:50.147190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:50.147192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:50.147216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:50.148726 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.149251 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.149254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.149256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.149360 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.149653 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.149655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.149656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.149820 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:50.149893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.150289 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.150401 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:16:50.151225 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.139964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42825 (local address 127.5.18.195:49827)
0928 21:16:50.140371 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.140412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.141597 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.142106 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.142113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.142441 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:50.142907 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.142920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.143676 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.143678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.144531 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.144540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.144996 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.145000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.145153 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.145681 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:50.145696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:50.147018 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:50.148829 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:50.148834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:50.148848 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:50.149167 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:50.149433 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:50.149435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:50.149436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:50.149558 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:50.149895 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:50.149899 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:50.150109 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.150245 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.150481 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260928 21:16:50.152179 6224 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42825
I20260928 21:16:50.152390 6224 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:50.152830 6224 heartbeater.cc:511] Master 127.5.18.254:42825 requested a full tablet report, sending...
I20260928 21:16:50.153743 5765 ts_manager.cc:194] Registered new tserver with Master: 18b682a6f1334135989080d4c2677b11 (127.5.18.195:33901)
I20260928 21:16:50.154260 5765 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:49827
I20260928 21:16:50.160691 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:50.170145 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.164085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.5.18.254:42825)
0928 21:16:50.164210 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.164215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.164342 (+ 127us) server_negotiation.cc:407] Connection header received
0928 21:16:50.164427 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.164429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.164464 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.164523 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:50.165088 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.165660 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.166423 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.166620 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.166688 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.167803 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:50.167806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:50.167808 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:50.167823 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:50.168878 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.169196 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.169198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.169199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.169260 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.169429 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.169431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.169432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.169665 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:50.169756 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.169917 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.169999 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:16:50.174115 5765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:50.176862 5765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:50.190696 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.186190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.5.18.194:43105)
0928 21:16:50.186461 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.186471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.186492 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:16:50.186559 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.186564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.186732 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.186876 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:50.187131 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.187924 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.189226 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.190057 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190164 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.190308 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.190446 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.190538 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260928 21:16:50.190728 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.184536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:33901 (local address 127.0.0.1:35876)
0928 21:16:50.184950 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.185025 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.185412 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.187169 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.187172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.187188 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:50.187280 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.187286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.188495 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.188498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.189291 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.189302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.190310 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.190403 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.190427 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.190475 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.190565 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260928 21:16:50.190732 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.186017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43105 (local address 127.0.0.1:43320)
0928 21:16:50.186219 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.186231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.186310 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.186941 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.186944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.186959 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:50.187036 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.187041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.188057 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.188060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.189114 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.189123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.190161 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.190267 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.190307 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.190411 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.190556 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":26,"thread_start_us":67,"threads_started":1}
I20260928 21:16:50.191301 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.185277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.5.18.195:33901)
0928 21:16:50.186759 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.186770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.186790 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:16:50.186855 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.186860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.187021 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.187197 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:50.187373 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.188387 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.189424 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.190192 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190900 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.191032 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.191078 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.191161 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":130,"threads_started":1}
I20260928 21:16:50.191431 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.184636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:44995 (local address 127.0.0.1:58384)
0928 21:16:50.185062 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.185073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.185178 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.186529 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.186535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.186558 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:50.186678 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.186684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.188109 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.188113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.189901 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.189909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.191073 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.191077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.191161 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.191182 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.191244 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.191317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":948,"thread_start_us":82,"threads_started":1}
I20260928 21:16:50.191495 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.185277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.5.18.193:44995)
0928 21:16:50.186060 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.186072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.186095 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:50.186172 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.186178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.186396 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.186517 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:50.186806 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.187963 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190005 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.190783 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.190880 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.191013 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.191278 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.191361 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":115,"threads_started":1}
I20260928 21:16:50.193944 6158 tablet_service.cc:1467] Processing CreateTablet for tablet 875c2d1abb66471b83444c8589569d82 (DEFAULT_TABLE table=test-table [id=7376ee7df2e644108bdcabc8c147a1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:50.193969 5890 tablet_service.cc:1467] Processing CreateTablet for tablet 875c2d1abb66471b83444c8589569d82 (DEFAULT_TABLE table=test-table [id=7376ee7df2e644108bdcabc8c147a1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:50.193969 6024 tablet_service.cc:1467] Processing CreateTablet for tablet 875c2d1abb66471b83444c8589569d82 (DEFAULT_TABLE table=test-table [id=7376ee7df2e644108bdcabc8c147a1aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:50.194864 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875c2d1abb66471b83444c8589569d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:50.194918 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875c2d1abb66471b83444c8589569d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:50.194918 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 875c2d1abb66471b83444c8589569d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:50.201159 6241 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: Bootstrap starting.
I20260928 21:16:50.201886 6243 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Bootstrap starting.
I20260928 21:16:50.202276 6242 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: Bootstrap starting.
I20260928 21:16:50.203444 6241 tablet_bootstrap.cc:654] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:50.204702 6242 tablet_bootstrap.cc:654] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:50.204715 6241 log.cc:824] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:50.205521 6242 log.cc:824] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:50.206804 6243 tablet_bootstrap.cc:654] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:50.206833 6241 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: No bootstrap required, opened a new log
I20260928 21:16:50.206997 6241 ts_tablet_manager.cc:1397] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 21:16:50.207280 6242 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: No bootstrap required, opened a new log
I20260928 21:16:50.207427 6242 ts_tablet_manager.cc:1397] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:50.208015 6243 log.cc:824] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:50.209851 6243 tablet_bootstrap.cc:492] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: No bootstrap required, opened a new log
I20260928 21:16:50.210024 6243 ts_tablet_manager.cc:1397] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 21:16:50.210435 6242 raft_consensus.cc:348] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.210646 6242 raft_consensus.cc:374] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:50.210721 6242 raft_consensus.cc:729] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f537c208d84bf9af08a73663f783cb, State: Initialized, Role: FOLLOWER
I20260928 21:16:50.210551 6241 raft_consensus.cc:348] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.210850 6241 raft_consensus.cc:374] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:50.210909 6241 raft_consensus.cc:729] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b682a6f1334135989080d4c2677b11, State: Initialized, Role: FOLLOWER
I20260928 21:16:50.211179 6242 consensus_queue.cc:260] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.211324 6241 consensus_queue.cc:260] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.211755 6090 heartbeater.cc:503] Master 127.5.18.254:42825 was elected leader, sending a full tablet report...
I20260928 21:16:50.212049 6242 ts_tablet_manager.cc:1428] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:16:50.212260 6241 ts_tablet_manager.cc:1428] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260928 21:16:50.212556 6224 heartbeater.cc:503] Master 127.5.18.254:42825 was elected leader, sending a full tablet report...
I20260928 21:16:50.213959 6243 raft_consensus.cc:348] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.214138 6243 raft_consensus.cc:374] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:50.214211 6243 raft_consensus.cc:729] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bce05fc2ee64a49ae1838cf25a84da2, State: Initialized, Role: FOLLOWER
I20260928 21:16:50.214603 6243 consensus_queue.cc:260] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.215195 5956 heartbeater.cc:503] Master 127.5.18.254:42825 was elected leader, sending a full tablet report...
I20260928 21:16:50.215443 6243 ts_tablet_manager.cc:1428] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:50.234500 6247 raft_consensus.cc:482] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:50.234674 6247 raft_consensus.cc:504] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.235778 6247 leader_election.cc:288] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18b682a6f1334135989080d4c2677b11 (127.5.18.195:33901), 0bce05fc2ee64a49ae1838cf25a84da2 (127.5.18.193:44995)
W20260928 21:16:50.237938 5957 tablet.cc:2369] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:50.239812 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.236068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:44995 (local address 127.5.18.194:33319)
0928 21:16:50.236368 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.236379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.236453 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.236654 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.236683 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.236703 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:50.236859 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.236865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.237885 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.237889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.238781 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.238791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.239418 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.239421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.239519 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.239541 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.239603 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.239657 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":63,"thread_start_us":103,"threads_started":1}
I20260928 21:16:50.239821 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.236176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:33319 (local address 127.5.18.193:44995)
0928 21:16:50.236309 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.236313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.236441 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:16:50.236523 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.236526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.236568 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.236639 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:50.236946 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.237741 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.238894 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.239299 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.239398 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.239443 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.239633 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.239708 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:16:50.239859 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.236059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45413 (local address 127.5.18.195:33901)
0928 21:16:50.236176 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.236181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.236365 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:16:50.236517 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.236520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.236567 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.236638 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:50.236946 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.237741 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.238983 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.239377 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.239483 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.239530 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.239695 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.239761 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:16:50.239849 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.235981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:33901 (local address 127.5.18.194:45413)
0928 21:16:50.236298 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:50.236312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:50.236410 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:50.236693 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:50.236695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:50.236715 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:50.236844 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.236851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.237858 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.237862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.238862 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:50.238874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.239491 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.239494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:50.239579 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.239599 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:16:50.239671 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:16:50.239732 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":52,"spinlock_wait_cycles":2304}
I20260928 21:16:50.240556 6178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875c2d1abb66471b83444c8589569d82" candidate_uuid: "57f537c208d84bf9af08a73663f783cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b682a6f1334135989080d4c2677b11" is_pre_election: true
I20260928 21:16:50.240551 5910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875c2d1abb66471b83444c8589569d82" candidate_uuid: "57f537c208d84bf9af08a73663f783cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" is_pre_election: true
I20260928 21:16:50.240790 5910 raft_consensus.cc:2454] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f537c208d84bf9af08a73663f783cb in term 0.
I20260928 21:16:50.240800 6178 raft_consensus.cc:2454] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57f537c208d84bf9af08a73663f783cb in term 0.
I20260928 21:16:50.241218 5979 leader_election.cc:302] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bce05fc2ee64a49ae1838cf25a84da2, 18b682a6f1334135989080d4c2677b11, 57f537c208d84bf9af08a73663f783cb; no voters:
I20260928 21:16:50.241475 6247 raft_consensus.cc:2781] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:50.241545 6247 raft_consensus.cc:482] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:50.241600 6247 raft_consensus.cc:3037] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:50.242537 6247 raft_consensus.cc:504] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.242839 6247 leader_election.cc:288] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [CANDIDATE]: Term 1 election: Requested vote from peers 18b682a6f1334135989080d4c2677b11 (127.5.18.195:33901), 0bce05fc2ee64a49ae1838cf25a84da2 (127.5.18.193:44995)
I20260928 21:16:50.243130 5910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875c2d1abb66471b83444c8589569d82" candidate_uuid: "57f537c208d84bf9af08a73663f783cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bce05fc2ee64a49ae1838cf25a84da2"
I20260928 21:16:50.243151 6178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "875c2d1abb66471b83444c8589569d82" candidate_uuid: "57f537c208d84bf9af08a73663f783cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18b682a6f1334135989080d4c2677b11"
I20260928 21:16:50.243232 5910 raft_consensus.cc:3037] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:50.243286 6178 raft_consensus.cc:3037] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:50.244284 5910 raft_consensus.cc:2454] T 875c2d1abb66471b83444c8589569d82 P 0bce05fc2ee64a49ae1838cf25a84da2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f537c208d84bf9af08a73663f783cb in term 1.
I20260928 21:16:50.244407 6178 raft_consensus.cc:2454] T 875c2d1abb66471b83444c8589569d82 P 18b682a6f1334135989080d4c2677b11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57f537c208d84bf9af08a73663f783cb in term 1.
I20260928 21:16:50.244536 5979 leader_election.cc:302] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bce05fc2ee64a49ae1838cf25a84da2, 57f537c208d84bf9af08a73663f783cb; no voters:
I20260928 21:16:50.244690 6247 raft_consensus.cc:2781] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:50.244853 6247 raft_consensus.cc:686] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [term 1 LEADER]: Becoming Leader. State: Replica: 57f537c208d84bf9af08a73663f783cb, State: Running, Role: LEADER
I20260928 21:16:50.245141 6247 consensus_queue.cc:237] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } }
I20260928 21:16:50.247905 5764 catalog_manager.cc:5495] T 875c2d1abb66471b83444c8589569d82 P 57f537c208d84bf9af08a73663f783cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 57f537c208d84bf9af08a73663f783cb (127.5.18.194). New cstate: current_term: 1 leader_uuid: "57f537c208d84bf9af08a73663f783cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18b682a6f1334135989080d4c2677b11" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bce05fc2ee64a49ae1838cf25a84da2" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57f537c208d84bf9af08a73663f783cb" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43105 } health_report { overall_health: HEALTHY } } }
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:50.268014 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.260432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.5.18.193:44995)
0928 21:16:50.260554 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.260559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.260680 (+ 121us) server_negotiation.cc:407] Connection header received
0928 21:16:50.260776 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.260778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.260819 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.260881 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:50.261609 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.262115 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.262896 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.263096 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.263162 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.265292 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:50.265302 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:50.265322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:50.265344 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:50.266675 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.267038 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.267042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.267044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.267123 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.267296 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.267298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.267299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.267523 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:50.267664 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.267777 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.267864 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:50.291729 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.284242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.5.18.193:44995)
0928 21:16:50.284369 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.284373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.284521 (+ 148us) server_negotiation.cc:407] Connection header received
0928 21:16:50.284650 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.284654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.284698 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.284761 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:50.285396 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.285951 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.286736 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.286950 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.287032 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.289090 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:50.289093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:50.289095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:50.289117 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:50.290284 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.290639 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.290643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.290645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.290714 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.290948 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.290952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.290953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.291106 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:50.291216 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.291475 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.291583 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[5720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:50.314661 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:50.306914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.5.18.193:44995)
0928 21:16:50.307016 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:50.307019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:50.307230 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:16:50.307376 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:50.307380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:50.307425 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:50.307488 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:50.308178 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.308724 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.309513 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:50.309792 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:50.309878 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:50.312010 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:50.312014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:50.312017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:50.312042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:50.313278 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.313642 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.313646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.313647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.313718 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:50.313946 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:50.313950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:50.313951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:50.314103 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:50.314213 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:50.314415 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:16:50.314513 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:16:50.315909 5870 tablet_service.cc:2908] Rejecting scan request for tablet 875c2d1abb66471b83444c8589569d82: Uninitialized: clean time has not yet been initialized
I20260928 21:16:50.317811 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5827
I20260928 21:16:50.322664 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5961
I20260928 21:16:50.326895 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6095
I20260928 21:16:50.331034 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 5733
2026-09-28T21:16:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2116 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[6265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[6265](info): set up 2 sockets
Sep 28 21:16:50 dist-test-slave-rc9f krb5kdc[6265](info): commencing operation
krb5kdc: starting...
W20260928 21:16:51.268965 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:16:51.380525 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:33321
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:33321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:51.482616 6278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:51.482883 6278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:51.482939 6278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:51.486874 6278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:51.486932 6278 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:16:51.486958 6278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:51.486979 6278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:51.487004 6278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:51.487032 6278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:51.492128 6278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:33321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:33321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:51.493095 6278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:51.494087 6278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:51.502178 6284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:51.502173 6286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:51.502368 6278 server_base.cc:1034] running on GCE node
W20260928 21:16:51.502173 6283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:51.502959 6278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:51.503952 6278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:51.505101 6278 hybrid_clock.cc:648] HybridClock initialized: now 1790630211505085 us; error 30 us; skew 500 ppm
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:51.512437 6278 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:16:51.513442 6278 webserver.cc:460] Webserver started at http://127.5.18.254:34567/ using document root <none> and password file <none>
I20260928 21:16:51.514002 6278 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:51.514060 6278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:51.514222 6278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:51.516165 6278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "b838057aadd84987b40671013bb65ae5"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "bb3719ba3351ee5d38c0044c948b9b00"
server_key_iv: "574391884a40ae3946fe8689449884b2"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.516635 6278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "b838057aadd84987b40671013bb65ae5"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "bb3719ba3351ee5d38c0044c948b9b00"
server_key_iv: "574391884a40ae3946fe8689449884b2"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.520037 6278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:16:51.522213 6293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:51.523356 6278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:51.523453 6278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "b838057aadd84987b40671013bb65ae5"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "bb3719ba3351ee5d38c0044c948b9b00"
server_key_iv: "574391884a40ae3946fe8689449884b2"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.523543 6278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:51.542358 6278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:51.545315 6278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:51.545482 6278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:51.552685 6278 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:33321
I20260928 21:16:51.552692 6355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:33321 every 8 connection(s)
I20260928 21:16:51.553596 6278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:51.555052 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6278
I20260928 21:16:51.555142 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:16:51.555380 5195 external_mini_cluster.cc:1428] Setting key 911d3390197bc47712ea2e66bea1b12a
I20260928 21:16:51.556394 6356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:51.562417 6356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Bootstrap starting.
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:51.564836 6356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:51.565487 6356 log.cc:824] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:51.567394 6356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: No bootstrap required, opened a new log
I20260928 21:16:51.569006 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.556680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.5.18.254:33321)
0928 21:16:51.557050 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:51.557065 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:51.557100 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:51.557756 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:51.557774 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:51.558086 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:51.558350 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:51.559170 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.560117 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.561219 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.561697 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.561806 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:51.564336 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:51.564340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:51.564353 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:51.564377 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:51.566486 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.566942 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.566946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.566950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.567048 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.567312 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.567314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.567315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.567670 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:51.567808 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:51.568092 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 21:16:51.568345 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260928 21:16:51.570333 6356 raft_consensus.cc:348] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } }
I20260928 21:16:51.570533 6356 raft_consensus.cc:374] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:51.570588 6356 raft_consensus.cc:729] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b838057aadd84987b40671013bb65ae5, State: Initialized, Role: FOLLOWER
I20260928 21:16:51.571012 6356 consensus_queue.cc:260] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } }
I20260928 21:16:51.571143 6356 raft_consensus.cc:388] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:51.571203 6356 raft_consensus.cc:482] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:51.571300 6356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:51.572055 6356 raft_consensus.cc:504] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } }
I20260928 21:16:51.572358 6356 leader_election.cc:302] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b838057aadd84987b40671013bb65ae5; no voters:
I20260928 21:16:51.572615 6356 leader_election.cc:288] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:51.572702 6361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:51.572893 6361 raft_consensus.cc:686] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [term 1 LEADER]: Becoming Leader. State: Replica: b838057aadd84987b40671013bb65ae5, State: Running, Role: LEADER
I20260928 21:16:51.573189 6361 consensus_queue.cc:237] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } }
I20260928 21:16:51.573720 6356 sys_catalog.cc:564] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:51.574688 6363 sys_catalog.cc:455] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b838057aadd84987b40671013bb65ae5. Latest consensus state: current_term: 1 leader_uuid: "b838057aadd84987b40671013bb65ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } } }
I20260928 21:16:51.574818 6363 sys_catalog.cc:458] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:51.575155 6370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:51.575109 6362 sys_catalog.cc:455] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b838057aadd84987b40671013bb65ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b838057aadd84987b40671013bb65ae5" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33321 } } }
I20260928 21:16:51.575676 6362 sys_catalog.cc:458] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:51.578109 6370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:51.583206 6370 catalog_manager.cc:1347] Generated new cluster ID: 3cadb36752544daeaa33b5bf100c2c5a
I20260928 21:16:51.583304 6370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:51.599846 6370 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:51.601034 6370 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:51.610687 6370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Generated new TSK 0
I20260928 21:16:51.611452 6370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:16:51.637811 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:51.742959 6382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:51.743206 6382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:51.743252 6382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:51.747190 6382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:51.747243 6382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:51.747291 6382 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:51.747341 6382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:16:51.752468 6382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:51.753430 6382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:51.754426 6382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:51.763374 6390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:51.763370 6387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:51.763392 6388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:51.763523 6382 server_base.cc:1034] running on GCE node
I20260928 21:16:51.763971 6382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:51.764544 6382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:51.765681 6382 hybrid_clock.cc:648] HybridClock initialized: now 1790630211765651 us; error 45 us; skew 500 ppm
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:51.768297 6382 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:16:51.769101 6382 webserver.cc:460] Webserver started at http://127.5.18.193:34153/ using document root <none> and password file <none>
I20260928 21:16:51.769639 6382 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:51.769706 6382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:51.769909 6382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:51.771826 6382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "81a4a0f01b4341e38e3e1019504b4158"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "31a80809b76391d9ed161046d02e3810"
server_key_iv: "08403dba7800869b7378e34d48fa0c2a"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.772341 6382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "81a4a0f01b4341e38e3e1019504b4158"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "31a80809b76391d9ed161046d02e3810"
server_key_iv: "08403dba7800869b7378e34d48fa0c2a"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.775789 6382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:16:51.777999 6397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:51.779093 6382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:51.779184 6382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "81a4a0f01b4341e38e3e1019504b4158"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "31a80809b76391d9ed161046d02e3810"
server_key_iv: "08403dba7800869b7378e34d48fa0c2a"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.779274 6382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:51.794657 6382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:51.797319 6382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:51.797487 6382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:51.798794 6382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:51.798856 6382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:51.798893 6382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:51.798919 6382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:51.808169 6382 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:38509
I20260928 21:16:51.808265 6512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:38509 every 8 connection(s)
I20260928 21:16:51.809122 6382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:51.811792 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.799490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:53917 (local address 127.5.18.254:33321)
0928 21:16:51.799632 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:51.799637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:51.800399 (+ 762us) server_negotiation.cc:407] Connection header received
0928 21:16:51.801472 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:51.801476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:51.801532 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:51.801606 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:51.802885 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.803603 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.804698 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.805051 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.805159 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:51.808414 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:51.808417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:51.808419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:51.808443 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:51.809994 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.810434 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.810438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.810440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.810514 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.810810 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.810814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.810816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.810983 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:51.811070 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:51.811510 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 21:16:51.811635 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:16:51.812489 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.799828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.193:53917)
0928 21:16:51.800254 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:51.800299 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:51.801224 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:51.801923 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:51.801928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:51.802218 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:51.802736 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:51.802756 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.803720 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.803724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:51.804583 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:51.804592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.805204 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.805209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:51.805422 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:51.806604 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:51.806624 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:51.808249 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:51.810091 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:51.810096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:51.810110 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:51.810346 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:51.810591 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:51.810594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:51.810596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:51.810720 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:51.811077 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:51.811083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:51.811299 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 21:16:51.811452 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:16:51.811745 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260928 21:16:51.813058 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6382
I20260928 21:16:51.813167 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:51.813410 5195 external_mini_cluster.cc:1428] Setting key 1b8222239d49bbf3c73c3a6cfa04123a
I20260928 21:16:51.813541 6513 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33321
I20260928 21:16:51.813776 6513 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:51.814268 6513 heartbeater.cc:511] Master 127.5.18.254:33321 requested a full tablet report, sending...
I20260928 21:16:51.815649 6309 ts_manager.cc:194] Registered new tserver with Master: 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509)
I20260928 21:16:51.817025 6309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:53917
I20260928 21:16:51.822036 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.815703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57189 (local address 127.5.18.254:33321)
0928 21:16:51.815803 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:51.815806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:51.816004 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:16:51.816098 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:51.816101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:51.816140 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:51.816200 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:51.816609 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.817308 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.818222 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.818444 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.818525 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:51.819437 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:51.819441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:51.819443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:51.819464 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:51.820670 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.821023 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.821026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.821028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.821097 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:51.821327 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:51.821330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:51.821332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:51.821481 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:51.821559 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:51.821804 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 21:16:51.821901 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:51.822050 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.815650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.193:57189)
0928 21:16:51.815933 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:51.815943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:51.816022 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:51.816221 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:51.816224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:51.816397 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:51.816511 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:51.816517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.817428 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.817430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:51.818122 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:51.818130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:51.818535 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:51.818539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:51.818622 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:51.819031 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:51.819044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:51.819324 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:51.820761 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:51.820764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:51.820767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:51.820944 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:51.821172 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:51.821175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:51.821177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:51.821249 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:51.821582 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:51.821585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:51.821664 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 21:16:51.821746 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:16:51.821876 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:16:51.837612 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:51.941830 6522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:51.942062 6522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:51.942109 6522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:51.946015 6522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:51.946067 6522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:51.946112 6522 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:51.946164 6522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:16:51.951295 6522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:51.952346 6522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:51.953313 6522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:51.962168 6528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:51.962174 6527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:51.962241 6530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:51.962388 6522 server_base.cc:1034] running on GCE node
I20260928 21:16:51.962815 6522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:51.963380 6522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:51.964527 6522 hybrid_clock.cc:648] HybridClock initialized: now 1790630211964500 us; error 39 us; skew 500 ppm
Sep 28 21:16:51 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:51.967080 6522 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:16:51.967852 6522 webserver.cc:460] Webserver started at http://127.5.18.194:41237/ using document root <none> and password file <none>
I20260928 21:16:51.968331 6522 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:51.968384 6522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:51.968542 6522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:51.970455 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "5ac4a29d122f4bb19ac4d41dea5bea62"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "b62da59da7661cbe89b5552270e57489"
server_key_iv: "673bc7037282d3cad9302bc20b9e0328"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.970916 6522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "5ac4a29d122f4bb19ac4d41dea5bea62"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "b62da59da7661cbe89b5552270e57489"
server_key_iv: "673bc7037282d3cad9302bc20b9e0328"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.974291 6522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:51.976406 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:51.977495 6522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:51.977622 6522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "5ac4a29d122f4bb19ac4d41dea5bea62"
format_stamp: "Formatted at 2026-09-28 21:16:51 on dist-test-slave-rc9f"
server_key: "b62da59da7661cbe89b5552270e57489"
server_key_iv: "673bc7037282d3cad9302bc20b9e0328"
server_key_version: "encryptionkey@0"
I20260928 21:16:51.977712 6522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:51.992830 6522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:51.995815 6522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:51.995980 6522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:51.997336 6522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:51.997385 6522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:51.997435 6522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:51.997464 6522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:52.006886 6522 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43865
I20260928 21:16:52.007239 6652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43865 every 8 connection(s)
I20260928 21:16:52.007867 6522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:52.009845 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.998006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43179 (local address 127.5.18.254:33321)
0928 21:16:51.998140 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:51.998145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:51.998992 (+ 847us) server_negotiation.cc:407] Connection header received
0928 21:16:51.999992 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:51.999995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.000053 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.000114 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.001677 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.002261 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.003590 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.003815 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.003906 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.006584 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.006588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.006591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.006618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.008036 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.008568 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.008572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.008574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.008662 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.008941 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.008943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.008944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.009100 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.009168 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.009613 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.009727 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:16:52.010587 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:51.998402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.194:43179)
0928 21:16:51.998831 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:51.998878 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:51.999752 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.000525 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.000534 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.000882 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:52.001520 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.001541 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.002383 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.002386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.003477 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.003487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.003966 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.003970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.004160 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.004731 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:52.004746 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:52.006420 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:52.008142 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.008147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.008161 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.008472 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.008749 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.008753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.008755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.008881 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.009172 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:52.009177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:52.009397 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.009544 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.009829 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260928 21:16:52.011610 6653 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33321
I20260928 21:16:52.011739 6653 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:52.012287 6653 heartbeater.cc:511] Master 127.5.18.254:33321 requested a full tablet report, sending...
I20260928 21:16:52.012887 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6522
I20260928 21:16:52.012997 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:52.013280 5195 external_mini_cluster.cc:1428] Setting key 9c078fb78d4c3694a39f7f085acf5ea3
I20260928 21:16:52.013381 6310 ts_manager.cc:194] Registered new tserver with Master: 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194:43865)
I20260928 21:16:52.014117 6310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:43179
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:16:52.022161 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.013824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:54553 (local address 127.5.18.254:33321)
0928 21:16:52.013928 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.013932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.014547 (+ 615us) server_negotiation.cc:407] Connection header received
0928 21:16:52.015914 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.015917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.015963 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.016025 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.016414 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.017026 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.017889 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.018213 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.018330 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.019243 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.019248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.019250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.019275 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.020675 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.021023 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.021026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.021029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.021114 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.021331 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.021334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.021336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.021605 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.021699 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.021924 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.022020 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:16:52.022176 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.013765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.194:54553)
0928 21:16:52.014479 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.014492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.014607 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.016052 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.016055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.016226 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:52.016313 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.016318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.017134 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.017136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.017758 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.017768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.018325 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.018328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.018391 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.018816 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:52.018829 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:52.019118 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:52.020767 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.020770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.020771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.020936 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.021191 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.021193 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.021195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.021256 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.021682 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:52.021684 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:52.021770 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.021870 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.022002 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:16:52.038000 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:52.143765 6662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:52.144003 6662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:52.144047 6662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:52.147943 6662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:52.148000 6662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:52.148038 6662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:52.148088 6662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:16:52.153110 6662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:33321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:52.154076 6662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:52.155012 6662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:52.163937 6667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:52.163991 6662 server_base.cc:1034] running on GCE node
W20260928 21:16:52.163941 6668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:52.163923 6670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:52.164572 6662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:52.165122 6662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:52.166276 6662 hybrid_clock.cc:648] HybridClock initialized: now 1790630212166259 us; error 32 us; skew 500 ppm
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630212, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:52.168751 6662 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:16:52.169521 6662 webserver.cc:460] Webserver started at http://127.5.18.195:46215/ using document root <none> and password file <none>
I20260928 21:16:52.170037 6662 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:52.170094 6662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:52.170256 6662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:52.172144 6662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "926cf9f890274b3bb4876d0ca4209d0a"
format_stamp: "Formatted at 2026-09-28 21:16:52 on dist-test-slave-rc9f"
server_key: "79715325d95cb6b37933f1d6a428ba5f"
server_key_iv: "03962ea51a81cde598e09df88c334a10"
server_key_version: "encryptionkey@0"
I20260928 21:16:52.172632 6662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "926cf9f890274b3bb4876d0ca4209d0a"
format_stamp: "Formatted at 2026-09-28 21:16:52 on dist-test-slave-rc9f"
server_key: "79715325d95cb6b37933f1d6a428ba5f"
server_key_iv: "03962ea51a81cde598e09df88c334a10"
server_key_version: "encryptionkey@0"
I20260928 21:16:52.176254 6662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:16:52.178519 6677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:52.179625 6662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:52.179721 6662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "926cf9f890274b3bb4876d0ca4209d0a"
format_stamp: "Formatted at 2026-09-28 21:16:52 on dist-test-slave-rc9f"
server_key: "79715325d95cb6b37933f1d6a428ba5f"
server_key_iv: "03962ea51a81cde598e09df88c334a10"
server_key_version: "encryptionkey@0"
I20260928 21:16:52.179816 6662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:52.195047 6662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:52.197701 6662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:52.197854 6662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:52.199167 6662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:52.199211 6662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:52.199260 6662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:52.199288 6662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:52.208551 6662 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:40581
I20260928 21:16:52.208662 6792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:40581 every 8 connection(s)
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630212, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:52.209540 6662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:52.212468 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.199955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:34881 (local address 127.5.18.254:33321)
0928 21:16:52.200065 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.200069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.201039 (+ 970us) server_negotiation.cc:407] Connection header received
0928 21:16:52.202103 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.202107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.202167 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.202224 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.203829 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.204615 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.206275 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.206493 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.206575 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.209383 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.209386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.209388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.209413 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.210830 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.211224 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.211226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.211228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.211297 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.211531 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.211533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.211534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.211687 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.211772 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.212253 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.212352 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:16:52.213008 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6662
I20260928 21:16:52.213096 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:52.213268 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.200460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.195:34881)
0928 21:16:52.200899 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.200938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.201863 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.202632 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.202641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.203043 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:52.203679 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.203700 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.204754 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.204758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.206152 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.206162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.206641 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.206646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.206860 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.207486 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:52.207506 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:52.209207 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:52.210918 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.210922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.210931 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.211152 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.211355 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.211357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.211358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.211470 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.211787 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:52.211793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:52.212040 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.212189 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.212415 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260928 21:16:52.213364 5195 external_mini_cluster.cc:1428] Setting key 535b790ff3769c995319dbfc8e029075
I20260928 21:16:52.215026 6793 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33321
I20260928 21:16:52.215205 6793 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:52.215678 6793 heartbeater.cc:511] Master 127.5.18.254:33321 requested a full tablet report, sending...
I20260928 21:16:52.216588 6308 ts_manager.cc:194] Registered new tserver with Master: 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581)
I20260928 21:16:52.217229 6308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:34881
I20260928 21:16:52.222802 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.216374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.5.18.195:43539)
0928 21:16:52.216604 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.216615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.216704 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.216923 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.216925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.217077 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:52.217163 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.217168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.217992 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.217995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.218604 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.218613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.219031 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.219035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.219118 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.219569 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:52.219581 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:52.219891 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:52.221399 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.221402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.221404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.221622 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.221870 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.221873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.221875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.221954 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.222297 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:52.222300 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:52.222387 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.222469 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.222609 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260928 21:16:52.222796 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.216405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:43539 (local address 127.5.18.254:33321)
0928 21:16:52.216496 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.216500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.216684 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:16:52.216792 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.216795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.216837 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.216917 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.217252 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.217875 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.218710 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.218937 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.219019 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.220014 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.220018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.220020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.220044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.221298 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.221705 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.221708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.221710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.221793 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.222028 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.222031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.222033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.222204 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.222287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.222532 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.222636 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 21:16:52.227831 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:52.251434 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.244108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.5.18.254:33321)
0928 21:16:52.244231 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.244235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.244248 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:52.244290 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.244292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.244331 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.244398 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.245068 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.245637 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.246470 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.246691 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.246772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.248936 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.248939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.248941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.248966 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.250117 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.250467 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.250469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.250470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.250551 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.250740 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.250742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.250743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.250900 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.250996 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.251179 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.251282 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:16:52.254227 6308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:52.257141 6308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:52.271420 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.266027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:40581 (local address 127.0.0.1:54570)
0928 21:16:52.266248 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.266261 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.266343 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.267251 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.267254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.267270 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.267355 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.267361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268787 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.268791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.269741 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.269749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270955 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.271120 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.271141 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.271193 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.271251 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":60,"thread_start_us":98,"threads_started":1}
I20260928 21:16:52.271468 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.266306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.5.18.195:40581)
0928 21:16:52.266832 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.266843 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.266866 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:52.266932 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.266938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.267109 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.267481 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.267599 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268644 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.269847 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270818 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.271048 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.271208 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.271258 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.271347 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":109,"threads_started":1}
I20260928 21:16:52.271631 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.266767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.5.18.193:38509)
0928 21:16:52.267043 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.267054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.267077 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:52.267151 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.267156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.267351 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.267507 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.267716 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268432 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270085 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270689 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270789 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.270958 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.271330 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.271424 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260928 21:16:52.271901 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.266047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.5.18.194:43865)
0928 21:16:52.266489 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.266501 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.266523 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:16:52.266603 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.266609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.266827 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.267319 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.267450 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268604 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270070 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270818 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.271490 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.271637 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.271681 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.271755 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":101,"threads_started":1}
I20260928 21:16:52.271418 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.265204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43865 (local address 127.0.0.1:52182)
0928 21:16:52.265557 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.265669 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.265797 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.266993 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.266999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.267021 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.267161 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.267168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268723 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.268726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.269945 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.269956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270955 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.271079 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.271100 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.271148 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.271240 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260928 21:16:52.272195 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.266330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:38509 (local address 127.0.0.1:35696)
0928 21:16:52.266524 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.266535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.266604 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.267488 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.267491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.267506 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.267616 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.267622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.268536 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.268539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.269945 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.269956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.270807 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.270811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.270922 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.270962 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.271067 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.271198 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260928 21:16:52.274654 6587 tablet_service.cc:1467] Processing CreateTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 (DEFAULT_TABLE table=test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.274893 6447 tablet_service.cc:1467] Processing CreateTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 (DEFAULT_TABLE table=test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.274742 6727 tablet_service.cc:1467] Processing CreateTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 (DEFAULT_TABLE table=test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.275658 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b19db2abf4199974e6db1b0ff1ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.275754 6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b19db2abf4199974e6db1b0ff1ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.275761 6727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b19db2abf4199974e6db1b0ff1ab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.282819 6816 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Bootstrap starting.
I20260928 21:16:52.283450 6818 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Bootstrap starting.
I20260928 21:16:52.283617 6817 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Bootstrap starting.
I20260928 21:16:52.284924 6816 tablet_bootstrap.cc:654] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.285581 6818 tablet_bootstrap.cc:654] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.285589 6817 tablet_bootstrap.cc:654] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.285624 6816 log.cc:824] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:52.286382 6818 log.cc:824] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:52.286382 6817 log.cc:824] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:52.287438 6816 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: No bootstrap required, opened a new log
I20260928 21:16:52.287622 6816 ts_tablet_manager.cc:1397] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:16:52.288210 6818 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: No bootstrap required, opened a new log
I20260928 21:16:52.288378 6818 ts_tablet_manager.cc:1397] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:16:52.288553 6817 tablet_bootstrap.cc:492] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: No bootstrap required, opened a new log
I20260928 21:16:52.288735 6817 ts_tablet_manager.cc:1397] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:16:52.291221 6818 raft_consensus.cc:348] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.291428 6818 raft_consensus.cc:374] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.291495 6818 raft_consensus.cc:729] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926cf9f890274b3bb4876d0ca4209d0a, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.291330 6816 raft_consensus.cc:348] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.291551 6816 raft_consensus.cc:374] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.291631 6816 raft_consensus.cc:729] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac4a29d122f4bb19ac4d41dea5bea62, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.291967 6818 consensus_queue.cc:260] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.292065 6816 consensus_queue.cc:260] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.292346 6817 raft_consensus.cc:348] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.292546 6793 heartbeater.cc:503] Master 127.5.18.254:33321 was elected leader, sending a full tablet report...
I20260928 21:16:52.292637 6817 raft_consensus.cc:374] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.292734 6817 raft_consensus.cc:729] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a4a0f01b4341e38e3e1019504b4158, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.292907 6818 ts_tablet_manager.cc:1428] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 21:16:52.292977 6816 ts_tablet_manager.cc:1428] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:16:52.293159 6817 consensus_queue.cc:260] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.293334 6653 heartbeater.cc:503] Master 127.5.18.254:33321 was elected leader, sending a full tablet report...
I20260928 21:16:52.293810 6513 heartbeater.cc:503] Master 127.5.18.254:33321 was elected leader, sending a full tablet report...
I20260928 21:16:52.294036 6817 ts_tablet_manager.cc:1428] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 21:16:52.310326 6514 tablet.cc:2369] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:52.322937 6823 raft_consensus.cc:482] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:52.323122 6823 raft_consensus.cc:504] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.324194 6823 leader_election.cc:288] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581), 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509)
I20260928 21:16:52.328051 6546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.324424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:40581 (local address 127.5.18.194:45917)
0928 21:16:52.324546 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.324562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.324662 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.324937 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.324940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.324959 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.325074 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.325080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.325960 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.325964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.327068 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.327076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.327689 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.327693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.327785 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.327809 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.327885 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.327956 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 21:16:52.328112 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.324515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45917 (local address 127.5.18.195:40581)
0928 21:16:52.324633 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.324638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.324651 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:52.324792 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.324796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.324849 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.324928 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.325208 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.325841 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.327178 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.327571 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.327675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.327717 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.327917 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.327996 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:16:52.328687 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b19db2abf4199974e6db1b0ff1ab7" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a" is_pre_election: true
I20260928 21:16:52.328928 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.324771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43427 (local address 127.5.18.193:38509)
0928 21:16:52.324866 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.324870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.325036 (+ 166us) server_negotiation.cc:407] Connection header received
0928 21:16:52.325149 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.325152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.325200 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.325285 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.325961 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.326727 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.327886 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.328433 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.328532 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.328566 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.328760 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.328817 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:16:52.328934 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.324672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:38509 (local address 127.5.18.194:43427)
0928 21:16:52.324960 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.324972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.325060 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.325296 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.325298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.325313 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.325819 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.325828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.326843 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.326846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.327778 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.327786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.328555 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.328558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.328655 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.328676 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.328751 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.328814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":136,"threads_started":1}
I20260928 21:16:52.329012 6747 raft_consensus.cc:2454] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 0.
I20260928 21:16:52.329445 6538 leader_election.cc:302] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac4a29d122f4bb19ac4d41dea5bea62, 926cf9f890274b3bb4876d0ca4209d0a; no voters:
I20260928 21:16:52.329668 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b19db2abf4199974e6db1b0ff1ab7" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a4a0f01b4341e38e3e1019504b4158" is_pre_election: true
I20260928 21:16:52.329699 6823 raft_consensus.cc:2781] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:52.329768 6823 raft_consensus.cc:482] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:52.329813 6823 raft_consensus.cc:3037] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.329914 6467 raft_consensus.cc:2454] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 0.
I20260928 21:16:52.330839 6823 raft_consensus.cc:504] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.331128 6823 leader_election.cc:288] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 election: Requested vote from peers 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581), 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509)
I20260928 21:16:52.331431 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b19db2abf4199974e6db1b0ff1ab7" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a"
I20260928 21:16:52.331431 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b19db2abf4199974e6db1b0ff1ab7" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a4a0f01b4341e38e3e1019504b4158"
I20260928 21:16:52.331542 6467 raft_consensus.cc:3037] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.331565 6747 raft_consensus.cc:3037] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.332713 6467 raft_consensus.cc:2454] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 1.
I20260928 21:16:52.332967 6538 leader_election.cc:302] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac4a29d122f4bb19ac4d41dea5bea62, 81a4a0f01b4341e38e3e1019504b4158; no voters:
I20260928 21:16:52.332990 6747 raft_consensus.cc:2454] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 1.
I20260928 21:16:52.333127 6823 raft_consensus.cc:2781] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:52.333310 6823 raft_consensus.cc:686] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 LEADER]: Becoming Leader. State: Replica: 5ac4a29d122f4bb19ac4d41dea5bea62, State: Running, Role: LEADER
I20260928 21:16:52.333644 6823 consensus_queue.cc:237] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } }
I20260928 21:16:52.336385 6308 catalog_manager.cc:5495] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:52.353322 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.349818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.5.18.194:43865)
0928 21:16:52.349927 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.349932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.350039 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:16:52.350297 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.350300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.350373 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.350473 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:52.350756 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.351353 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.352159 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.352364 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.352432 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.352673 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.353037 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.353115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.353163 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.353210 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:16:52.358791 6747 raft_consensus.cc:1261] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Refusing update from remote peer 5ac4a29d122f4bb19ac4d41dea5bea62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.358791 6467 raft_consensus.cc:1261] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Refusing update from remote peer 5ac4a29d122f4bb19ac4d41dea5bea62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.359490 6823 consensus_queue.cc:1035] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.359709 6826 consensus_queue.cc:1035] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.371045 6830 mvcc.cc:205] Tried to move back new op lower bound from 7334421349813768192 to 7334421349720895488. Current Snapshot: MvccSnapshot[applied={T|T < 7334421349813768192}]
I20260928 21:16:52.377377 6308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:52.377532 6308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:52.379642 6308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Sending DeleteTablet for 3 replicas of tablet f50b19db2abf4199974e6db1b0ff1ab7
I20260928 21:16:52.380308 6587 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:52182
I20260928 21:16:52.380615 6447 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:35696
I20260928 21:16:52.380755 6727 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b19db2abf4199974e6db1b0ff1ab7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:54570
I20260928 21:16:52.381068 6840 tablet_replica.cc:330] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: stopping tablet replica
I20260928 21:16:52.381223 6839 tablet_replica.cc:330] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: stopping tablet replica
I20260928 21:16:52.381421 6840 raft_consensus.cc:2229] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:52.381489 6839 raft_consensus.cc:2229] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:52.381511 6308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:16:52.381726 6840 raft_consensus.cc:2258] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:52.381829 6839 raft_consensus.cc:2258] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:16:52.381922 6308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:52.383147 6839 ts_tablet_manager.cc:1905] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.383282 6841 tablet_replica.cc:330] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: stopping tablet replica
I20260928 21:16:52.383669 6841 raft_consensus.cc:2229] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:52.384018 6841 raft_consensus.cc:2258] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:52.384156 6840 ts_tablet_manager.cc:1905] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.385512 6841 ts_tablet_manager.cc:1905] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.386736 6840 ts_tablet_manager.cc:1918] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:52.386830 6840 log.cc:1196] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/f50b19db2abf4199974e6db1b0ff1ab7
I20260928 21:16:52.388137 6841 ts_tablet_manager.cc:1918] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:52.388290 6841 log.cc:1196] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/f50b19db2abf4199974e6db1b0ff1ab7
I20260928 21:16:52.388537 6727 tablet_service.cc:1467] Processing CreateTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f (DEFAULT_TABLE table=test-table [id=8935c509a22d4a16b3ce335031c557f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.388615 6841 ts_tablet_manager.cc:1939] T f50b19db2abf4199974e6db1b0ff1ab7 P 926cf9f890274b3bb4876d0ca4209d0a: Deleting consensus metadata
I20260928 21:16:52.388811 6727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18122911e1b04aaeb3da7e85ef74d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.388901 6447 tablet_service.cc:1467] Processing CreateTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f (DEFAULT_TABLE table=test-table [id=8935c509a22d4a16b3ce335031c557f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.389137 6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18122911e1b04aaeb3da7e85ef74d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.391108 6840 ts_tablet_manager.cc:1939] T f50b19db2abf4199974e6db1b0ff1ab7 P 81a4a0f01b4341e38e3e1019504b4158: Deleting consensus metadata
I20260928 21:16:52.391244 6817 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Bootstrap starting.
I20260928 21:16:52.391434 6587 tablet_service.cc:1467] Processing CreateTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f (DEFAULT_TABLE table=test-table [id=8935c509a22d4a16b3ce335031c557f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:52.391690 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18122911e1b04aaeb3da7e85ef74d27f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.392156 6817 tablet_bootstrap.cc:654] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.393596 6817 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: No bootstrap required, opened a new log
I20260928 21:16:52.393672 6817 ts_tablet_manager.cc:1397] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:52.393719 6818 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Bootstrap starting.
I20260928 21:16:52.394068 6817 raft_consensus.cc:348] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.394172 6817 raft_consensus.cc:374] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.394200 6817 raft_consensus.cc:729] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a4a0f01b4341e38e3e1019504b4158, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.394289 6817 consensus_queue.cc:260] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.394500 6818 tablet_bootstrap.cc:654] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.394522 6817 ts_tablet_manager.cc:1428] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:16:52.394697 6816 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Bootstrap starting.
I20260928 21:16:52.395360 6294 catalog_manager.cc:4841] TS 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509): tablet f50b19db2abf4199974e6db1b0ff1ab7 (table test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]) successfully deleted
I20260928 21:16:52.395538 6816 tablet_bootstrap.cc:654] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.395583 6294 catalog_manager.cc:4841] TS 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581): tablet f50b19db2abf4199974e6db1b0ff1ab7 (table test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]) successfully deleted
I20260928 21:16:52.397444 6839 ts_tablet_manager.cc:1918] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:52.397526 6839 log.cc:1196] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/f50b19db2abf4199974e6db1b0ff1ab7
I20260928 21:16:52.397833 6839 ts_tablet_manager.cc:1939] T f50b19db2abf4199974e6db1b0ff1ab7 P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting consensus metadata
I20260928 21:16:52.398928 6294 catalog_manager.cc:4841] TS 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194:43865): tablet f50b19db2abf4199974e6db1b0ff1ab7 (table test-table [id=8cc22d3092a94f0f96da50b3b51a49e6]) successfully deleted
I20260928 21:16:52.402311 6818 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: No bootstrap required, opened a new log
I20260928 21:16:52.402400 6818 ts_tablet_manager.cc:1397] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 21:16:52.402561 6816 tablet_bootstrap.cc:492] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: No bootstrap required, opened a new log
I20260928 21:16:52.402644 6816 ts_tablet_manager.cc:1397] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260928 21:16:52.402873 6818 raft_consensus.cc:348] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.402974 6818 raft_consensus.cc:374] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.403013 6818 raft_consensus.cc:729] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926cf9f890274b3bb4876d0ca4209d0a, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.403050 6816 raft_consensus.cc:348] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.403167 6816 raft_consensus.cc:374] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.403139 6818 consensus_queue.cc:260] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.403203 6816 raft_consensus.cc:729] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac4a29d122f4bb19ac4d41dea5bea62, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.403307 6816 consensus_queue.cc:260] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.403405 6818 ts_tablet_manager.cc:1428] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:52.403537 6816 ts_tablet_manager.cc:1428] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 21:16:52.460677 6794 tablet.cc:2369] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:52.508909 6654 tablet.cc:2369] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:52.571678 6824 raft_consensus.cc:482] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:52.571805 6824 raft_consensus.cc:504] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.572818 6824 leader_election.cc:288] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194:43865), 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581)
I20260928 21:16:52.576916 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.573545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:53867 (local address 127.5.18.195:40581)
0928 21:16:52.573681 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.573684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.573855 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:16:52.573942 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.573945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.573988 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.574056 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.574304 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.574880 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.575951 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.576317 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.576403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.576452 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.576722 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.576789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:16:52.576917 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.573127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:54129 (local address 127.5.18.194:43865)
0928 21:16:52.573286 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.573291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.573407 (+ 116us) server_negotiation.cc:407] Connection header received
0928 21:16:52.573613 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.573616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.573668 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.573745 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:52.574059 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.574757 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.575953 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.576351 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.576463 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.576514 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.576726 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.576794 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:16:52.576947 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.573069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43865 (local address 127.5.18.193:54129)
0928 21:16:52.573299 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.573312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.573436 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.573768 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.573770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.573804 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.573945 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.573952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.574861 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.574864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.575838 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.575847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.576460 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.576464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.576612 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.576631 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.576694 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.576792 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":109,"spinlock_wait_cycles":15872,"thread_start_us":79,"threads_started":1}
I20260928 21:16:52.576956 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.573433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:40581 (local address 127.5.18.193:53867)
0928 21:16:52.573737 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.573750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.573865 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.574069 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.574071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.574087 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:52.574201 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.574207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.574989 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.574991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.575845 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.575854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.576436 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.576439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.576588 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.576628 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.576688 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.576794 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":59,"spinlock_wait_cycles":768,"thread_start_us":78,"threads_started":1}
I20260928 21:16:52.577435 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18122911e1b04aaeb3da7e85ef74d27f" candidate_uuid: "81a4a0f01b4341e38e3e1019504b4158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" is_pre_election: true
I20260928 21:16:52.577518 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18122911e1b04aaeb3da7e85ef74d27f" candidate_uuid: "81a4a0f01b4341e38e3e1019504b4158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a" is_pre_election: true
I20260928 21:16:52.577673 6747 raft_consensus.cc:2454] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a4a0f01b4341e38e3e1019504b4158 in term 0.
I20260928 21:16:52.577677 6601 raft_consensus.cc:2454] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a4a0f01b4341e38e3e1019504b4158 in term 0.
I20260928 21:16:52.578004 6398 leader_election.cc:302] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a4a0f01b4341e38e3e1019504b4158, 926cf9f890274b3bb4876d0ca4209d0a; no voters:
I20260928 21:16:52.578205 6824 raft_consensus.cc:2781] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:52.578258 6824 raft_consensus.cc:482] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:52.578310 6824 raft_consensus.cc:3037] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.579219 6824 raft_consensus.cc:504] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.579526 6824 leader_election.cc:288] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [CANDIDATE]: Term 1 election: Requested vote from peers 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194:43865), 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581)
I20260928 21:16:52.579839 6601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18122911e1b04aaeb3da7e85ef74d27f" candidate_uuid: "81a4a0f01b4341e38e3e1019504b4158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62"
I20260928 21:16:52.579854 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18122911e1b04aaeb3da7e85ef74d27f" candidate_uuid: "81a4a0f01b4341e38e3e1019504b4158" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a"
I20260928 21:16:52.579934 6747 raft_consensus.cc:3037] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.579936 6601 raft_consensus.cc:3037] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.580771 6747 raft_consensus.cc:2454] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a4a0f01b4341e38e3e1019504b4158 in term 1.
I20260928 21:16:52.580937 6601 raft_consensus.cc:2454] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a4a0f01b4341e38e3e1019504b4158 in term 1.
I20260928 21:16:52.581015 6398 leader_election.cc:302] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a4a0f01b4341e38e3e1019504b4158, 926cf9f890274b3bb4876d0ca4209d0a; no voters:
I20260928 21:16:52.581279 6824 raft_consensus.cc:2781] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:52.581460 6824 raft_consensus.cc:686] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 1 LEADER]: Becoming Leader. State: Replica: 81a4a0f01b4341e38e3e1019504b4158, State: Running, Role: LEADER
I20260928 21:16:52.581730 6824 consensus_queue.cc:237] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.583734 6308 catalog_manager.cc:5495] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193). New cstate: current_term: 1 leader_uuid: "81a4a0f01b4341e38e3e1019504b4158" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:52.654525 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.647121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33321 (local address 127.0.0.1:58580)
0928 21:16:52.647392 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.647404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.647512 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.647730 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.647733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.648149 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:52.648241 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.648248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.648948 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.648951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.649492 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.649499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.649942 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.649945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.650065 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.650531 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:52.650543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:52.651896 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:52.653197 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.653204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.653210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.653433 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.653682 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:52.653684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:52.653686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:52.653769 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:52.654060 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:52.654065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:52.654166 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.654252 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.654364 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":19,"thread_start_us":107,"threads_started":1}
I20260928 21:16:52.654537 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.647182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.5.18.254:33321)
0928 21:16:52.647316 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.647320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.647477 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:16:52.647595 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.647598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.647635 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.647696 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.648356 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.648853 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.649614 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.649848 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.649932 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.652024 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.652026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.652028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.652050 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.653105 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.653501 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.653503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.653505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.653590 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.653832 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.653834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.653835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.653989 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.654058 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.654303 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.654412 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":45}
I20260928 21:16:52.657968 6308 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58580:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:16:52.658655 6308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:52.662919 6727 tablet_service.cc:1467] Processing CreateTablet for tablet 9084dcb1e5fa4212b1d97f4e98226773 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b5ab621e7bc4e6aafe4e65459480b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:52.662911 6447 tablet_service.cc:1467] Processing CreateTablet for tablet 9084dcb1e5fa4212b1d97f4e98226773 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b5ab621e7bc4e6aafe4e65459480b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:52.662897 6587 tablet_service.cc:1467] Processing CreateTablet for tablet 9084dcb1e5fa4212b1d97f4e98226773 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b5ab621e7bc4e6aafe4e65459480b79]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:52.663287 6727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9084dcb1e5fa4212b1d97f4e98226773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.663287 6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9084dcb1e5fa4212b1d97f4e98226773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.663292 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9084dcb1e5fa4212b1d97f4e98226773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:52.665586 6816 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: Bootstrap starting.
I20260928 21:16:52.666029 6818 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: Bootstrap starting.
I20260928 21:16:52.666452 6816 tablet_bootstrap.cc:654] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.666944 6818 tablet_bootstrap.cc:654] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.667389 6817 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: Bootstrap starting.
I20260928 21:16:52.667454 6816 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: No bootstrap required, opened a new log
I20260928 21:16:52.667529 6816 ts_tablet_manager.cc:1397] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:52.667925 6816 raft_consensus.cc:348] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.668035 6816 raft_consensus.cc:374] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.668069 6816 raft_consensus.cc:729] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac4a29d122f4bb19ac4d41dea5bea62, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.668159 6818 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: No bootstrap required, opened a new log
I20260928 21:16:52.668179 6816 consensus_queue.cc:260] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.668244 6818 ts_tablet_manager.cc:1397] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:52.668265 6817 tablet_bootstrap.cc:654] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:52.668411 6816 ts_tablet_manager.cc:1428] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:52.668576 6836 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.668675 6836 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.668665 6818 raft_consensus.cc:348] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.668771 6818 raft_consensus.cc:374] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.668812 6818 raft_consensus.cc:729] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926cf9f890274b3bb4876d0ca4209d0a, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.668926 6818 consensus_queue.cc:260] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.669169 6818 ts_tablet_manager.cc:1428] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:52.669395 6822 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.669541 6822 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.669855 6817 tablet_bootstrap.cc:492] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: No bootstrap required, opened a new log
I20260928 21:16:52.669929 6817 ts_tablet_manager.cc:1397] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:52.670287 6817 raft_consensus.cc:348] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.670395 6817 raft_consensus.cc:374] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:52.670428 6817 raft_consensus.cc:729] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a4a0f01b4341e38e3e1019504b4158, State: Initialized, Role: FOLLOWER
I20260928 21:16:52.670533 6817 consensus_queue.cc:260] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.670794 6817 ts_tablet_manager.cc:1428] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:52.670753 6824 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.670881 6824 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.701248 6836 raft_consensus.cc:482] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:52.701360 6836 raft_consensus.cc:504] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.701752 6836 leader_election.cc:288] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509), 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581)
I20260928 21:16:52.702154 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9084dcb1e5fa4212b1d97f4e98226773" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a" is_pre_election: true
I20260928 21:16:52.702138 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9084dcb1e5fa4212b1d97f4e98226773" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a4a0f01b4341e38e3e1019504b4158" is_pre_election: true
I20260928 21:16:52.702292 6747 raft_consensus.cc:2454] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 0.
I20260928 21:16:52.702297 6467 raft_consensus.cc:2454] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 0.
I20260928 21:16:52.702553 6538 leader_election.cc:302] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac4a29d122f4bb19ac4d41dea5bea62, 81a4a0f01b4341e38e3e1019504b4158; no voters:
I20260928 21:16:52.702740 6836 raft_consensus.cc:2781] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:52.702808 6836 raft_consensus.cc:482] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:52.702836 6836 raft_consensus.cc:3037] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.703756 6836 raft_consensus.cc:504] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.704046 6836 leader_election.cc:288] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 election: Requested vote from peers 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509), 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581)
I20260928 21:16:52.704316 6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9084dcb1e5fa4212b1d97f4e98226773" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81a4a0f01b4341e38e3e1019504b4158"
I20260928 21:16:52.704345 6747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9084dcb1e5fa4212b1d97f4e98226773" candidate_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926cf9f890274b3bb4876d0ca4209d0a"
I20260928 21:16:52.704408 6467 raft_consensus.cc:3037] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.704434 6747 raft_consensus.cc:3037] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:52.705240 6467 raft_consensus.cc:2454] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 1.
I20260928 21:16:52.705257 6747 raft_consensus.cc:2454] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ac4a29d122f4bb19ac4d41dea5bea62 in term 1.
I20260928 21:16:52.705487 6538 leader_election.cc:302] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac4a29d122f4bb19ac4d41dea5bea62, 81a4a0f01b4341e38e3e1019504b4158; no voters:
I20260928 21:16:52.705657 6836 raft_consensus.cc:2781] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:52.705745 6836 raft_consensus.cc:686] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 LEADER]: Becoming Leader. State: Replica: 5ac4a29d122f4bb19ac4d41dea5bea62, State: Running, Role: LEADER
I20260928 21:16:52.705873 6836 consensus_queue.cc:237] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } }
I20260928 21:16:52.706454 6826 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: TxnStatusTablet state changed. Reason: New leader 5ac4a29d122f4bb19ac4d41dea5bea62. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.706542 6826 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:52.706727 6859 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:16:52.706909 6859 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:16:52.707370 6308 catalog_manager.cc:5495] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:52.720625 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.717416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43865 (local address 127.0.0.1:52212)
0928 21:16:52.717546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.717577 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.717672 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.717896 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.717898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.717904 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:52.717986 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.717990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.718905 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.718907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.719555 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.719564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.720053 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.720056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.720113 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.720152 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.720403 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.720414 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.720461 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.720524 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:16:52.720647 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.717472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.5.18.194:43865)
0928 21:16:52.717658 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.717663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.717677 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:52.717762 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.717764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.717815 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.717889 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:52.718063 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.718824 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.719646 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.719971 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.720061 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.720249 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.720331 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.720392 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.720481 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.720528 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 21:16:52.722622 6747 raft_consensus.cc:1261] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Refusing update from remote peer 5ac4a29d122f4bb19ac4d41dea5bea62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.722708 6467 raft_consensus.cc:1261] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Refusing update from remote peer 5ac4a29d122f4bb19ac4d41dea5bea62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.722978 6826 consensus_queue.cc:1035] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.723397 6826 consensus_queue.cc:1035] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.724041 6822 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: TxnStatusTablet state changed. Reason: New leader 5ac4a29d122f4bb19ac4d41dea5bea62. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.724134 6822 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.724570 6824 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: TxnStatusTablet state changed. Reason: New leader 5ac4a29d122f4bb19ac4d41dea5bea62. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.724676 6824 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.726248 6829 mvcc.cc:205] Tried to move back new op lower bound from 7334421351308595200 to 7334421351244390400. Current Snapshot: MvccSnapshot[applied={T|T < 7334421351308595200}]
I20260928 21:16:52.727245 6836 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.727357 6836 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:52.728897 6824 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.728991 6824 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 81a4a0f01b4341e38e3e1019504b4158: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.729908 6823 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.730002 6823 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 5ac4a29d122f4bb19ac4d41dea5bea62: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:52.730134 6822 tablet_replica.cc:439] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81a4a0f01b4341e38e3e1019504b4158" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 38509 } } peers { permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 } } peers { permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 } } }
I20260928 21:16:52.730216 6822 tablet_replica.cc:442] T 9084dcb1e5fa4212b1d97f4e98226773 P 926cf9f890274b3bb4876d0ca4209d0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:52.730521 6831 mvcc.cc:205] Tried to move back new op lower bound from 7334421351308595200 to 7334421351244390400. Current Snapshot: MvccSnapshot[applied={T|T < 7334421351308595200}]
I20260928 21:16:52.736233 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.732708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.5.18.193:38509)
0928 21:16:52.732827 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.732831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.732847 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:16:52.732973 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.732976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.733032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.733126 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:52.733312 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.734082 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.734814 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.735140 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.735226 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.735412 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.735916 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.736018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.736047 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.736096 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:16:52.746464 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.742741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43865 (local address 127.5.18.193:35613)
0928 21:16:52.742988 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.743000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.743076 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.743267 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.743268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.743274 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:52.743371 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.743374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.744250 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.744252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.745347 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.745356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.745891 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.745893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.745957 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.745992 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.746244 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.746247 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.746288 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.746345 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260928 21:16:52.746470 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.742775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:35613 (local address 127.5.18.194:43865)
0928 21:16:52.742880 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.742884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.743055 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:16:52.743137 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.743140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.743186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.743259 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:52.743445 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.744163 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.745443 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.745800 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.745895 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.746099 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.746170 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.746233 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.746304 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.746350 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:52.752467 6747 raft_consensus.cc:1261] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Refusing update from remote peer 81a4a0f01b4341e38e3e1019504b4158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.752696 6601 raft_consensus.cc:1261] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Refusing update from remote peer 81a4a0f01b4341e38e3e1019504b4158: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:52.752892 6824 consensus_queue.cc:1035] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926cf9f890274b3bb4876d0ca4209d0a" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.753155 6847 consensus_queue.cc:1035] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ac4a29d122f4bb19ac4d41dea5bea62" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:52.774325 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.771081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:36173 (local address 127.5.18.193:38509)
0928 21:16:52.771182 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.771187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.771377 (+ 190us) server_negotiation.cc:407] Connection header received
0928 21:16:52.771483 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.771487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.771530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.771587 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:52.771785 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.772328 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.773331 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.773543 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.773660 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.773924 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.773984 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.774031 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.774125 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.774170 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:16:52.774365 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.771022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:38509 (local address 127.5.18.194:36173)
0928 21:16:52.771306 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:52.771317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:52.771408 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:52.771599 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:52.771602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:52.771609 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:52.771700 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.771705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.772444 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.772447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.773229 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:52.773239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.773660 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.773664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:52.773759 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.773811 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:52.774050 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:52.774056 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:16:52.774113 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:16:52.774208 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260928 21:16:52.795204 6309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:52.795367 6309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:52.797326 6309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b838057aadd84987b40671013bb65ae5: Sending DeleteTablet for 3 replicas of tablet 18122911e1b04aaeb3da7e85ef74d27f
I20260928 21:16:52.797873 6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:35696
I20260928 21:16:52.798041 6840 tablet_replica.cc:330] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: stopping tablet replica
I20260928 21:16:52.798139 6840 raft_consensus.cc:2229] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:52.798239 6727 tablet_service.cc:1514] Processing DeleteTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:54570
I20260928 21:16:52.798393 6841 tablet_replica.cc:330] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: stopping tablet replica
I20260928 21:16:52.798389 6840 raft_consensus.cc:2258] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:52.798484 6841 raft_consensus.cc:2229] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:52.798576 6841 raft_consensus.cc:2258] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:52.798870 6587 tablet_service.cc:1514] Processing DeleteTablet for tablet 18122911e1b04aaeb3da7e85ef74d27f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:52 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:52182
I20260928 21:16:52.799196 6840 ts_tablet_manager.cc:1905] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.801676 6839 tablet_replica.cc:330] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: stopping tablet replica
I20260928 21:16:52.801806 6839 raft_consensus.cc:2229] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:52.801919 6839 raft_consensus.cc:2258] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:52.802479 6841 ts_tablet_manager.cc:1905] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.805058 6841 ts_tablet_manager.cc:1918] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:52.805135 6841 log.cc:1196] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/18122911e1b04aaeb3da7e85ef74d27f
I20260928 21:16:52.805368 6841 ts_tablet_manager.cc:1939] T 18122911e1b04aaeb3da7e85ef74d27f P 926cf9f890274b3bb4876d0ca4209d0a: Deleting consensus metadata
I20260928 21:16:52.806032 6294 catalog_manager.cc:4841] TS 926cf9f890274b3bb4876d0ca4209d0a (127.5.18.195:40581): tablet 18122911e1b04aaeb3da7e85ef74d27f (table test-table [id=8935c509a22d4a16b3ce335031c557f5]) successfully deleted
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:16:52.806751 6839 ts_tablet_manager.cc:1905] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:52.809389 6839 ts_tablet_manager.cc:1918] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:52.809468 6839 log.cc:1196] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/18122911e1b04aaeb3da7e85ef74d27f
I20260928 21:16:52.809720 6839 ts_tablet_manager.cc:1939] T 18122911e1b04aaeb3da7e85ef74d27f P 5ac4a29d122f4bb19ac4d41dea5bea62: Deleting consensus metadata
I20260928 21:16:52.810349 6294 catalog_manager.cc:4841] TS 5ac4a29d122f4bb19ac4d41dea5bea62 (127.5.18.194:43865): tablet 18122911e1b04aaeb3da7e85ef74d27f (table test-table [id=8935c509a22d4a16b3ce335031c557f5]) successfully deleted
I20260928 21:16:52.811177 6840 ts_tablet_manager.cc:1918] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:52.811249 6840 log.cc:1196] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/18122911e1b04aaeb3da7e85ef74d27f
I20260928 21:16:52.811465 6840 ts_tablet_manager.cc:1939] T 18122911e1b04aaeb3da7e85ef74d27f P 81a4a0f01b4341e38e3e1019504b4158: Deleting consensus metadata
I20260928 21:16:52.812067 6294 catalog_manager.cc:4841] TS 81a4a0f01b4341e38e3e1019504b4158 (127.5.18.193:38509): tablet 18122911e1b04aaeb3da7e85ef74d27f (table test-table [id=8935c509a22d4a16b3ce335031c557f5]) successfully deleted
I20260928 21:16:52.812285 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.799418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.5.18.193:38509)
0928 21:16:52.799520 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.799524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.799807 (+ 283us) server_negotiation.cc:407] Connection header received
0928 21:16:52.799964 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.799967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.800007 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.800050 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.800943 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.801701 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.802992 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.803288 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.803356 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.807317 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.807327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.807341 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.807366 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.809211 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.809891 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.809897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.809899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.810006 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.810330 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.810334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.810335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.810618 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.810777 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.811257 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.811385 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 21:16:52.812661 6507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35714
I20260928 21:16:52.823597 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:52.814907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58584 (local address 127.5.18.254:33321)
0928 21:16:52.815013 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:52.815017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:52.815412 (+ 395us) server_negotiation.cc:407] Connection header received
0928 21:16:52.815568 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:52.815571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:52.815634 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:52.815682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:52.816530 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.817328 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.818462 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:52.818793 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:52.818859 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:52.820363 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:52.820367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:52.820370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:52.820394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:52.821900 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.822349 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.822352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.822354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.822434 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:52.822703 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:52.822706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:52.822708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:52.822901 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:52.822996 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:52.823339 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 21:16:52.823446 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 21:16:52.824489 6309 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58584
I20260928 21:16:52.825914 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6382
I20260928 21:16:52.832482 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6522
I20260928 21:16:52.837925 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6662
I20260928 21:16:52.842101 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6278
2026-09-28T21:16:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2511 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6881](info): set up 2 sockets
Sep 28 21:16:52 dist-test-slave-rc9f krb5kdc[6881](info): commencing operation
krb5kdc: starting...
W20260928 21:16:53.785008 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:16:53 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:16:53.896045 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:36773
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:54.000125 6894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:54.000399 6894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:54.000458 6894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:54.004392 6894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:54.004449 6894 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:16:54.004472 6894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:54.004494 6894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:54.004524 6894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:54.004546 6894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:54.009727 6894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:36773
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:54.010720 6894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:54.011685 6894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:54.019502 6900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:54.019623 6899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.019698 6894 server_base.cc:1034] running on GCE node
W20260928 21:16:54.019651 6902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.020309 6894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:54.021308 6894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:54.022452 6894 hybrid_clock.cc:648] HybridClock initialized: now 1790630214022435 us; error 28 us; skew 500 ppm
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:54.024875 6894 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:16:54.025681 6894 webserver.cc:460] Webserver started at http://127.5.18.254:38973/ using document root <none> and password file <none>
I20260928 21:16:54.026177 6894 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:54.026244 6894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:54.026414 6894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:54.028265 6894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "cf4be8b432c34d41ac5e758723cdc350"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "55a45221a22dcac073204a1b9cfaad43"
server_key_iv: "a4603d0878112a6db0d553991aefab68"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.028766 6894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "cf4be8b432c34d41ac5e758723cdc350"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "55a45221a22dcac073204a1b9cfaad43"
server_key_iv: "a4603d0878112a6db0d553991aefab68"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.032188 6894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 21:16:54.034338 6909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.035451 6894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 21:16:54.035553 6894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "cf4be8b432c34d41ac5e758723cdc350"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "55a45221a22dcac073204a1b9cfaad43"
server_key_iv: "a4603d0878112a6db0d553991aefab68"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.035637 6894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:54.053112 6894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:54.055688 6894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:54.055855 6894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:54.062805 6894 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:36773
I20260928 21:16:54.062851 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:36773 every 8 connection(s)
I20260928 21:16:54.063660 6894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:54.066186 6972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:54.071161 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6894
I20260928 21:16:54.071285 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:16:54.071545 5195 external_mini_cluster.cc:1428] Setting key 7f8e780b8807e0ea590a6031b6d08769
I20260928 21:16:54.071545 6972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Bootstrap starting.
I20260928 21:16:54.073952 6972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:54.074728 6972 log.cc:824] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:54.076539 6972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: No bootstrap required, opened a new log
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:54.080554 6972 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } }
I20260928 21:16:54.080799 6972 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:54.080855 6972 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4be8b432c34d41ac5e758723cdc350, State: Initialized, Role: FOLLOWER
I20260928 21:16:54.081300 6972 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } }
I20260928 21:16:54.081454 6972 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:54.081525 6972 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:54.081642 6972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:54.082436 6972 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } }
I20260928 21:16:54.082779 6972 leader_election.cc:302] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4be8b432c34d41ac5e758723cdc350; no voters:
I20260928 21:16:54.083101 6972 leader_election.cc:288] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:54.083176 6977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:54.083387 6977 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [term 1 LEADER]: Becoming Leader. State: Replica: cf4be8b432c34d41ac5e758723cdc350, State: Running, Role: LEADER
I20260928 21:16:54.083734 6977 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } }
I20260928 21:16:54.084332 6972 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:54.085314 6979 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf4be8b432c34d41ac5e758723cdc350. Latest consensus state: current_term: 1 leader_uuid: "cf4be8b432c34d41ac5e758723cdc350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } } }
I20260928 21:16:54.085420 6979 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:54.085425 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.072730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36720 (local address 127.5.18.254:36773)
0928 21:16:54.073103 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.073117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.073150 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:16:54.073792 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.073810 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.074141 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.074401 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.075204 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.076204 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.077300 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.077783 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.077894 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.080471 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.080476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.080488 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.080513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.082224 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.082652 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.082656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.082660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.082775 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.083049 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.083052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.083053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.083593 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.083821 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.084067 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.084479 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260928 21:16:54.085738 6978 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf4be8b432c34d41ac5e758723cdc350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4be8b432c34d41ac5e758723cdc350" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36773 } } }
I20260928 21:16:54.085815 6978 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:54.085956 6986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:54.088949 6986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:54.094594 6986 catalog_manager.cc:1347] Generated new cluster ID: 525412a3031d4571893444a8578b993f
I20260928 21:16:54.094650 6986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:54.108731 6986 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:54.109956 6986 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:54.119855 6986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Generated new TSK 0
I20260928 21:16:54.120515 6986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:16:54.152726 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36773
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:54.259013 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:54.259261 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:54.259317 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:54.263275 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:54.263336 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:54.263402 6998 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:16:54.263434 6998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:54.263494 6998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:16:54.268594 6998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:54.269629 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:54.270617 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:54.279651 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:54.279651 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.279758 6998 server_base.cc:1034] running on GCE node
W20260928 21:16:54.279645 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.280316 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:54.280917 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:54.282063 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790630214282038 us; error 38 us; skew 500 ppm
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:54.284564 6998 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:16:54.285444 6998 webserver.cc:460] Webserver started at http://127.5.18.193:35331/ using document root <none> and password file <none>
I20260928 21:16:54.286015 6998 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:54.286079 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:54.286275 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:54.288196 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "abbb916594374cacb8890b4478b070e0"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "47c9d0eaa91ae2aa642b61904bd18e94"
server_key_iv: "ecb46a10b089645e9ae0a2fd98711723"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.288717 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "abbb916594374cacb8890b4478b070e0"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "47c9d0eaa91ae2aa642b61904bd18e94"
server_key_iv: "ecb46a10b089645e9ae0a2fd98711723"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.292161 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:16:54.294363 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.295485 6998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:54.295593 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "abbb916594374cacb8890b4478b070e0"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "47c9d0eaa91ae2aa642b61904bd18e94"
server_key_iv: "ecb46a10b089645e9ae0a2fd98711723"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.295680 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:54.306185 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:54.309190 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:54.309362 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:54.310765 6998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:54.310812 6998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.310863 6998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:54.310891 6998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:54.320329 6998 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:35579
I20260928 21:16:54.320523 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:35579 every 8 connection(s)
I20260928 21:16:54.321346 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:54.324076 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.311525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:34303 (local address 127.5.18.254:36773)
0928 21:16:54.311630 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.311634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.312659 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 21:16:54.313716 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.313719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.313779 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.313836 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.315424 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.316225 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.317389 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.317707 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.317804 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.320565 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.320569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.320572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.320602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.322117 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.322591 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.322594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.322596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.322682 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.323002 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.323005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.323007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.323233 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.323329 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.323794 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.323916 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:16:54.324864 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.312063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.193:34303)
0928 21:16:54.312523 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.312563 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.313457 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.314245 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.314254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.314606 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.315253 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.315272 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.316334 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.316337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.317273 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.317283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.317885 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.317890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.318093 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.318749 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.318770 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.320367 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.322223 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.322228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.322244 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.322496 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.322786 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.322790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.322792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.322917 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.323339 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.323346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.323571 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.323735 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.324050 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":131,"threads_started":1}
I20260928 21:16:54.326249 7129 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36773
I20260928 21:16:54.326390 7129 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:54.326884 7129 heartbeater.cc:511] Master 127.5.18.254:36773 requested a full tablet report, sending...
I20260928 21:16:54.328114 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 6998
I20260928 21:16:54.328219 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:54.328421 6926 ts_manager.cc:194] Registered new tserver with Master: abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579)
I20260928 21:16:54.328495 5195 external_mini_cluster.cc:1428] Setting key 6de3fac08330c8804e014bba61fba4be
I20260928 21:16:54.329927 6926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:34303
I20260928 21:16:54.335247 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.328227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57927 (local address 127.5.18.254:36773)
0928 21:16:54.328333 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.328337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.328538 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:16:54.328632 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.328635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.328681 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.328761 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.329222 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.329967 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.330895 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.331183 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.331281 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.332449 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.332453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.332455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.332479 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.333829 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.334260 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.334262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.334264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.334325 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.334569 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.334587 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.334588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.334726 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.334794 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.335052 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.335136 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":42}
I20260928 21:16:54.335343 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.328149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.193:57927)
0928 21:16:54.328448 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.328462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.328559 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.328790 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.328792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.328987 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.329101 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.329109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.330148 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.330150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.330786 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.330796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.331296 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.331300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.331386 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.331895 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.331910 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.332306 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.333925 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.333929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.333931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.334172 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.334397 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.334400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.334402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.334494 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.334806 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.334809 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.334902 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.335018 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.335160 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:16:54.353539 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36773
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:54.457935 7138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:54.458180 7138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:54.458223 7138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:54.462106 7138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:54.462153 7138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:54.462189 7138 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:16:54.462208 7138 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:54.462237 7138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:16:54.467202 7138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:54.468060 7138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:54.468940 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:54.477634 7146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:54.477857 7143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.477805 7138 server_base.cc:1034] running on GCE node
W20260928 21:16:54.477654 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.478365 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:54.478899 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:54.480052 7138 hybrid_clock.cc:648] HybridClock initialized: now 1790630214480039 us; error 25 us; skew 500 ppm
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:54.482507 7138 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:16:54.483253 7138 webserver.cc:460] Webserver started at http://127.5.18.194:37597/ using document root <none> and password file <none>
I20260928 21:16:54.483732 7138 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:54.483793 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:54.483961 7138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:54.485935 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "9a196c427eb34630a1954d1aefc5c137"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "439a8355abfb33ae788266e69ff1bcd4"
server_key_iv: "a377910346f83e1c18fa01504c706f29"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.486471 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "9a196c427eb34630a1954d1aefc5c137"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "439a8355abfb33ae788266e69ff1bcd4"
server_key_iv: "a377910346f83e1c18fa01504c706f29"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.489909 7138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:54.492048 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.493147 7138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:54.493249 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "9a196c427eb34630a1954d1aefc5c137"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "439a8355abfb33ae788266e69ff1bcd4"
server_key_iv: "a377910346f83e1c18fa01504c706f29"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.493333 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:54.521361 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:54.524504 7138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:54.524652 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:54.526047 7138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:54.526103 7138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.526142 7138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:54.526171 7138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:54.535653 7138 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:38543
I20260928 21:16:54.535727 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:38543 every 8 connection(s)
I20260928 21:16:54.536542 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:54.538367 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.526688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:34861 (local address 127.5.18.254:36773)
0928 21:16:54.526834 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.526839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.528082 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 21:16:54.529242 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.529245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.529304 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.529398 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.530603 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.531164 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.532330 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.532545 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.532631 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.535171 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.535174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.535176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.535201 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.536568 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.537043 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.537046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.537048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.537131 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.537451 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.537454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.537455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.537681 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.537757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.538166 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.538255 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:16:54.538772 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7138
I20260928 21:16:54.538872 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:54.539098 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.527519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.194:34861)
0928 21:16:54.527944 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.527992 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.529019 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.529636 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.529645 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.530010 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.530465 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.530479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.531279 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.531283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.532221 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.532231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.532681 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.532684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.532864 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.533347 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.533362 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.535017 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.536671 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.536676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.536689 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.536931 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.537240 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.537243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.537245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.537362 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.537768 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.537772 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.537972 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.538110 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.538338 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260928 21:16:54.539113 5195 external_mini_cluster.cc:1428] Setting key 69b0a97f81d1198452a84cccb5db96fe
I20260928 21:16:54.540429 7269 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36773
I20260928 21:16:54.540556 7269 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:54.540982 7269 heartbeater.cc:511] Master 127.5.18.254:36773 requested a full tablet report, sending...
I20260928 21:16:54.541903 6926 ts_manager.cc:194] Registered new tserver with Master: 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543)
I20260928 21:16:54.542582 6926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:34861
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:16:54.549258 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.542339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.194:47453)
0928 21:16:54.542984 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.542999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.543120 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.543346 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.543348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.543523 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.543609 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.543614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.544299 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.544302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.544997 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.545006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.545527 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.545530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.545640 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.546049 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.546061 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.546345 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.547932 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.547935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.547936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.548115 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.548376 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.548378 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.548379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.548447 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.548863 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.548865 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.548936 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.549012 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.549129 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":93,"threads_started":1}
I20260928 21:16:54.549269 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.542403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:47453 (local address 127.5.18.254:36773)
0928 21:16:54.542503 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.542506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.543060 (+ 554us) server_negotiation.cc:407] Connection header received
0928 21:16:54.543221 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.543224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.543264 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.543322 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.543707 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.544191 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.545102 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.545424 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.545525 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.546466 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.546469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.546472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.546496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.547840 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.548212 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.548216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.548217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.548297 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.548515 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.548518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.548519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.548773 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.548878 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.549071 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.549155 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:16:54.564816 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36773
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:54.670449 7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:54.670672 7278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:54.670722 7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:54.674606 7278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:54.674654 7278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:54.674695 7278 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:16:54.674713 7278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:16:54.674743 7278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:16:54.679836 7278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:36773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:54.680727 7278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:54.681641 7278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:54.690562 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:54.690599 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.690667 7278 server_base.cc:1034] running on GCE node
W20260928 21:16:54.690572 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:54.691148 7278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:54.691708 7278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:54.692857 7278 hybrid_clock.cc:648] HybridClock initialized: now 1790630214692839 us; error 29 us; skew 500 ppm
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:54.695284 7278 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:16:54.696146 7278 webserver.cc:460] Webserver started at http://127.5.18.195:38177/ using document root <none> and password file <none>
I20260928 21:16:54.696655 7278 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:54.696725 7278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:54.696933 7278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:54.698887 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "9fba07043d9e39a33c0bc9098f8452fe"
server_key_iv: "3597aea179766bf523392edb2f11a1bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.699375 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "9fba07043d9e39a33c0bc9098f8452fe"
server_key_iv: "3597aea179766bf523392edb2f11a1bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.702734 7278 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:54.704833 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.705940 7278 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:54.706043 7278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
format_stamp: "Formatted at 2026-09-28 21:16:54 on dist-test-slave-rc9f"
server_key: "9fba07043d9e39a33c0bc9098f8452fe"
server_key_iv: "3597aea179766bf523392edb2f11a1bd"
server_key_version: "encryptionkey@0"
I20260928 21:16:54.706123 7278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:54.716811 7278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:54.719283 7278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:54.719440 7278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:54.720793 7278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:54.720839 7278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:54.720887 7278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:54.720916 7278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:54.730389 7408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:37355 every 8 connection(s)
I20260928 21:16:54.730225 7278 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:37355
I20260928 21:16:54.731469 7278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:54.733810 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.721398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:44717 (local address 127.5.18.254:36773)
0928 21:16:54.721539 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.721545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.722871 (+ 1326us) server_negotiation.cc:407] Connection header received
0928 21:16:54.724083 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.724086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.724134 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.724228 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.725711 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.726467 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.727720 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.727943 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.728030 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.730711 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.730714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.730716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.730743 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.732196 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.732644 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.732647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.732648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.732724 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.732963 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.732964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.732966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.733113 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.733180 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.733581 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.733691 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:16:54.734532 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.722194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.195:44717)
0928 21:16:54.722726 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.722777 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.723869 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.724420 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.724429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.724873 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.725540 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.725582 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.726584 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.726587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.727608 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.727618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.728072 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.728077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.728229 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.728858 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.728877 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.730567 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.732311 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.732316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.732329 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.732565 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.732791 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.732793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.732795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.732907 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.733179 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.733184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.733382 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.733517 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.733787 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":100,"threads_started":1}
I20260928 21:16:54.735562 7409 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36773
I20260928 21:16:54.735688 7409 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:54.736249 7409 heartbeater.cc:511] Master 127.5.18.254:36773 requested a full tablet report, sending...
I20260928 21:16:54.737355 6924 ts_manager.cc:194] Registered new tserver with Master: 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:54.738080 6924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:44717
I20260928 21:16:54.740537 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7278
I20260928 21:16:54.740638 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:54.740976 5195 external_mini_cluster.cc:1428] Setting key b5902d2e17b413891621e323a5ae78d4
I20260928 21:16:54.742868 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:54.744457 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.737912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56095 (local address 127.5.18.254:36773)
0928 21:16:54.738016 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.738020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.738232 (+ 212us) server_negotiation.cc:407] Connection header received
0928 21:16:54.738383 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.738386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.738416 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.738465 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.738921 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.739393 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.740425 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.740691 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.740774 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.741850 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.741853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.741855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.741875 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.743132 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.743473 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.743476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.743478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.743546 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.743772 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.743775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.743777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.743927 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.744006 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.744232 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.744323 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:16:54.744469 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.737845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.5.18.195:56095)
0928 21:16:54.738152 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.738164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.738293 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.738484 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.738486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.738648 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:54.738761 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.738769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.739521 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.739525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.740319 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.740331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.740787 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.740792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.740887 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.741343 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:54.741355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:54.741736 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:54.743222 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.743225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.743227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.743395 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.743621 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:54.743624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:54.743626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:54.743701 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:54.744008 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:54.744011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:54.744092 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.744172 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.744304 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":75,"threads_started":1}
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:16:54 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:54.765969 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.759015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.5.18.254:36773)
0928 21:16:54.759120 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.759123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.759134 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:16:54.759167 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.759169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.759206 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.759269 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:54.759850 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.760334 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.761093 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.761286 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.761347 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.763524 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:54.763527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:54.763529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:54.763551 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:54.764623 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.764984 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.764990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.764992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.765052 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:54.765256 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:54.765258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:54.765259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:54.765398 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:54.765486 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.765738 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.765822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:16:54.768373 6924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:54.771286 6924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:54.784571 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.778801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:35579 (local address 127.0.0.1:58162)
0928 21:16:54.779086 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.779131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.779244 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.780502 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.780505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.780521 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:54.780606 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.780611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.781974 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.781977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.783167 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.783176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.784008 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.784012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.784133 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784176 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.784300 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.784409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":24,"thread_start_us":58,"threads_started":1}
I20260928 21:16:54.784591 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.779285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.5.18.193:35579)
0928 21:16:54.780060 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.780072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.780095 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:54.780169 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.780174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.780370 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.780525 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:54.780697 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.781869 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.783279 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.783891 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.783989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784170 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.784331 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.784426 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":107,"threads_started":1}
I20260928 21:16:54.784795 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.779630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.5.18.194:38543)
0928 21:16:54.780002 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.780012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.780220 (+ 208us) server_negotiation.cc:407] Connection header received
0928 21:16:54.780415 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.780421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.780593 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.780722 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:54.780924 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.782000 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.783182 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.783961 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.784061 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784232 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.784504 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.784609 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":109,"threads_started":1}
I20260928 21:16:54.784868 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.779434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.5.18.195:37355)
0928 21:16:54.779806 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:54.779817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:54.779854 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:16:54.779930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:54.779935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:54.780149 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:54.780631 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:54.780762 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.781543 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.783160 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.783962 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.784491 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784630 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:54.784674 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 21:16:54.784749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":82,"threads_started":1}
I20260928 21:16:54.785187 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.779466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.0.0.1:53584)
0928 21:16:54.780135 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.780146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.780285 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.780720 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.780723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.780748 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:54.780825 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.780830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.782090 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.782094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.783061 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.783070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.784286 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.784289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.784365 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784384 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.784969 (+ 585us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.785029 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":87,"threads_started":1}
I20260928 21:16:54.785197 7427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:54.778910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37355 (local address 127.0.0.1:49460)
0928 21:16:54.779666 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:54.779680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:54.779784 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:54.780326 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:54.780332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:54.780353 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:54.780481 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.780488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.781652 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.781656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.783028 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:54.783039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:54.784092 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:54.784095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:54.784178 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:54.784215 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:16:54.784266 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:16:54.784389 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":98,"threads_started":1}
I20260928 21:16:54.788065 7343 tablet_service.cc:1467] Processing CreateTablet for tablet 755e83ae26aa49149d1607531a12c87d (DEFAULT_TABLE table=test-table [id=5b4238421db349cba3213a3003f4cb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:54.788055 7063 tablet_service.cc:1467] Processing CreateTablet for tablet 755e83ae26aa49149d1607531a12c87d (DEFAULT_TABLE table=test-table [id=5b4238421db349cba3213a3003f4cb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:54.789165 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e83ae26aa49149d1607531a12c87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:54.789170 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e83ae26aa49149d1607531a12c87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:54.790436 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 755e83ae26aa49149d1607531a12c87d (DEFAULT_TABLE table=test-table [id=5b4238421db349cba3213a3003f4cb0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:54.791226 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 755e83ae26aa49149d1607531a12c87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:54.796883 7432 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Bootstrap starting.
I20260928 21:16:54.797483 7433 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Bootstrap starting.
I20260928 21:16:54.797684 7434 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Bootstrap starting.
I20260928 21:16:54.799594 7434 tablet_bootstrap.cc:654] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:54.799644 7432 tablet_bootstrap.cc:654] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:54.799736 7433 tablet_bootstrap.cc:654] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:54.800438 7433 log.cc:824] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:54.800438 7432 log.cc:824] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:54.800439 7434 log.cc:824] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:54.802117 7432 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: No bootstrap required, opened a new log
I20260928 21:16:54.802119 7434 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: No bootstrap required, opened a new log
I20260928 21:16:54.802282 7434 ts_tablet_manager.cc:1397] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:16:54.802291 7432 ts_tablet_manager.cc:1397] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:54.803323 7433 tablet_bootstrap.cc:492] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: No bootstrap required, opened a new log
I20260928 21:16:54.803479 7433 ts_tablet_manager.cc:1397] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 21:16:54.805107 7434 raft_consensus.cc:348] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.805325 7434 raft_consensus.cc:374] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:54.805389 7434 raft_consensus.cc:729] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Initialized, Role: FOLLOWER
I20260928 21:16:54.805835 7434 consensus_queue.cc:260] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.805914 7432 raft_consensus.cc:348] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.806142 7432 raft_consensus.cc:374] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:54.806213 7432 raft_consensus.cc:729] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb916594374cacb8890b4478b070e0, State: Initialized, Role: FOLLOWER
I20260928 21:16:54.806588 7269 heartbeater.cc:503] Master 127.5.18.254:36773 was elected leader, sending a full tablet report...
I20260928 21:16:54.806634 7432 consensus_queue.cc:260] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.806807 7434 ts_tablet_manager.cc:1428] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260928 21:16:54.806988 7433 raft_consensus.cc:348] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.807205 7433 raft_consensus.cc:374] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:54.807250 7433 raft_consensus.cc:729] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Initialized, Role: FOLLOWER
I20260928 21:16:54.807512 7432 ts_tablet_manager.cc:1428] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:54.807806 7129 heartbeater.cc:503] Master 127.5.18.254:36773 was elected leader, sending a full tablet report...
I20260928 21:16:54.807603 7433 consensus_queue.cc:260] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:54.808957 7409 heartbeater.cc:503] Master 127.5.18.254:36773 was elected leader, sending a full tablet report...
I20260928 21:16:54.809303 7433 ts_tablet_manager.cc:1428] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 21:16:54.822443 7130 tablet.cc:2369] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:54.982319 7410 tablet.cc:2369] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:54.999040 7439 raft_consensus.cc:482] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:54.999280 7439 raft_consensus.cc:504] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:55.000401 7439 leader_election.cc:288] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355), 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543)
I20260928 21:16:55.004176 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.000770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57149 (local address 127.5.18.195:37355)
0928 21:16:55.000891 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.000895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.001211 (+ 316us) server_negotiation.cc:407] Connection header received
0928 21:16:55.001324 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.001326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.001375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.001446 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:55.001693 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.002176 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003283 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.003662 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.003824 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.003987 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.004055 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:16:55.004206 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.000762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:44527 (local address 127.5.18.194:38543)
0928 21:16:55.000890 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.000894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.001128 (+ 234us) server_negotiation.cc:407] Connection header received
0928 21:16:55.001250 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.001254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.001305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.001365 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:55.001653 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.002159 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003290 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.003668 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003773 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.003824 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.004017 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.004090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:16:55.004237 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.000630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37355 (local address 127.5.18.193:57149)
0928 21:16:55.001113 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.001127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.001238 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.001463 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.001466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.001479 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:55.001590 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.001596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.002278 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.002281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.003165 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.003173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.003773 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.003887 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.003905 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.003965 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.004032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":243,"thread_start_us":81,"threads_started":1}
I20260928 21:16:55.004591 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.000630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.5.18.193:44527)
0928 21:16:55.001036 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.001048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.001151 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.001388 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.001391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.001407 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:55.001534 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.001541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.002262 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.002265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.003168 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.003177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.003774 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.003778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.003913 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.003930 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.003993 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.004061 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":97,"thread_start_us":68,"threads_started":1}
I20260928 21:16:55.004889 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e83ae26aa49149d1607531a12c87d" candidate_uuid: "abbb916594374cacb8890b4478b070e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" is_pre_election: true
I20260928 21:16:55.005131 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e83ae26aa49149d1607531a12c87d" candidate_uuid: "abbb916594374cacb8890b4478b070e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a196c427eb34630a1954d1aefc5c137" is_pre_election: true
I20260928 21:16:55.005190 7363 raft_consensus.cc:2454] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbb916594374cacb8890b4478b070e0 in term 0.
I20260928 21:16:55.005385 7223 raft_consensus.cc:2454] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbb916594374cacb8890b4478b070e0 in term 0.
I20260928 21:16:55.005662 7015 leader_election.cc:302] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:55.005941 7439 raft_consensus.cc:2781] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:55.006016 7439 raft_consensus.cc:482] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:55.006067 7439 raft_consensus.cc:3037] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.007017 7439 raft_consensus.cc:504] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:55.007329 7439 leader_election.cc:288] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [CANDIDATE]: Term 1 election: Requested vote from peers 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355), 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543)
I20260928 21:16:55.007627 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e83ae26aa49149d1607531a12c87d" candidate_uuid: "abbb916594374cacb8890b4478b070e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a196c427eb34630a1954d1aefc5c137"
I20260928 21:16:55.007643 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "755e83ae26aa49149d1607531a12c87d" candidate_uuid: "abbb916594374cacb8890b4478b070e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
I20260928 21:16:55.007735 7223 raft_consensus.cc:3037] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.007759 7363 raft_consensus.cc:3037] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.008829 7223 raft_consensus.cc:2454] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbb916594374cacb8890b4478b070e0 in term 1.
I20260928 21:16:55.009155 7017 leader_election.cc:302] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a196c427eb34630a1954d1aefc5c137, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:55.009351 7439 raft_consensus.cc:2781] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:55.009411 7363 raft_consensus.cc:2454] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbb916594374cacb8890b4478b070e0 in term 1.
I20260928 21:16:55.009545 7439 raft_consensus.cc:686] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 1 LEADER]: Becoming Leader. State: Replica: abbb916594374cacb8890b4478b070e0, State: Running, Role: LEADER
I20260928 21:16:55.009945 7439 consensus_queue.cc:237] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } }
I20260928 21:16:55.012943 6924 catalog_manager.cc:5495] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to abbb916594374cacb8890b4478b070e0 (127.5.18.193). New cstate: current_term: 1 leader_uuid: "abbb916594374cacb8890b4478b070e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } health_report { overall_health: UNKNOWN } } }
W20260928 21:16:55.037599 7270 tablet.cc:2369] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:16:55 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:55.102596 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.095171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.5.18.254:36773)
0928 21:16:55.095294 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.095300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.095474 (+ 174us) server_negotiation.cc:407] Connection header received
0928 21:16:55.095593 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.095597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.095633 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.095696 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:55.096376 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.096893 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.097657 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.097863 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.097946 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.100033 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:55.100036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:55.100039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:55.100059 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:55.101094 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:55.101487 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:55.101490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:55.101492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:55.101556 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:55.101811 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:55.101814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:55.101816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:55.101970 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:55.102050 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.102310 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.102455 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":43}
I20260928 21:16:55.102596 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.095119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36773 (local address 127.0.0.1:36738)
0928 21:16:55.095395 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.095407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.095514 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.095731 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.095734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.096170 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:55.096271 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.096278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.096983 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.096986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.097526 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.097532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.097959 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.097962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.098067 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.098560 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:55.098571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:55.099924 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:55.101183 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:55.101192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:55.101200 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:55.101404 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:55.101655 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:55.101658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:55.101660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:55.101741 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:55.102057 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:55.102063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:55.102166 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.102258 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.102402 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":14,"thread_start_us":111,"threads_started":1}
I20260928 21:16:55.106218 6924 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:36738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:16:55.106901 6924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:55.111341 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 1deafa1869cd4d7c857343e09a11a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c51dbfaf8d945a382f5519d59b9a29e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:55.111328 7063 tablet_service.cc:1467] Processing CreateTablet for tablet 1deafa1869cd4d7c857343e09a11a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c51dbfaf8d945a382f5519d59b9a29e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:55.111481 7343 tablet_service.cc:1467] Processing CreateTablet for tablet 1deafa1869cd4d7c857343e09a11a176 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c51dbfaf8d945a382f5519d59b9a29e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:16:55.111981 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1deafa1869cd4d7c857343e09a11a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.112005 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1deafa1869cd4d7c857343e09a11a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.112004 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1deafa1869cd4d7c857343e09a11a176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.114226 7434 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: Bootstrap starting.
I20260928 21:16:55.114741 7432 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: Bootstrap starting.
I20260928 21:16:55.115087 7434 tablet_bootstrap.cc:654] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.115945 7432 tablet_bootstrap.cc:654] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.116258 7434 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: No bootstrap required, opened a new log
I20260928 21:16:55.116341 7434 ts_tablet_manager.cc:1397] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:55.116567 7433 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: Bootstrap starting.
I20260928 21:16:55.116776 7434 raft_consensus.cc:348] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.116889 7434 raft_consensus.cc:374] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.116920 7434 raft_consensus.cc:729] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.117062 7432 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: No bootstrap required, opened a new log
I20260928 21:16:55.117041 7434 consensus_queue.cc:260] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.117139 7432 ts_tablet_manager.cc:1397] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:55.117273 7434 ts_tablet_manager.cc:1428] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:55.117470 7433 tablet_bootstrap.cc:654] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.117559 7432 raft_consensus.cc:348] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.117681 7432 raft_consensus.cc:374] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.117709 7432 raft_consensus.cc:729] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb916594374cacb8890b4478b070e0, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.117815 7432 consensus_queue.cc:260] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.117897 7438 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.118033 7438 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.118096 7432 ts_tablet_manager.cc:1428] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:55.118067 7439 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.118173 7439 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.120090 7433 tablet_bootstrap.cc:492] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: No bootstrap required, opened a new log
I20260928 21:16:55.120165 7433 ts_tablet_manager.cc:1397] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:16:55.120599 7433 raft_consensus.cc:348] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.120694 7433 raft_consensus.cc:374] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.120728 7433 raft_consensus.cc:729] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.120849 7433 consensus_queue.cc:260] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.121110 7433 ts_tablet_manager.cc:1428] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:16:55.121127 7440 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.121276 7440 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.171875 7438 raft_consensus.cc:482] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:55.171996 7438 raft_consensus.cc:504] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.173048 7438 leader_election.cc:288] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:55.176892 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.173640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43071 (local address 127.5.18.193:35579)
0928 21:16:55.173766 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.173769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.173780 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:16:55.173821 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.173823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.173864 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.173936 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:55.174205 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.174667 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.175928 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.176314 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.176414 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.176467 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.176700 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.176771 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:55.176894 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.173394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:60863 (local address 127.5.18.195:37355)
0928 21:16:55.173514 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.173519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.173633 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:16:55.173769 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.173771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.173810 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.173864 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:55.174141 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.174645 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.175920 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.176314 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.176419 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.176468 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.176699 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.176770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:16:55.176918 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.173273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37355 (local address 127.5.18.194:60863)
0928 21:16:55.173554 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.173586 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.173673 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.173889 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.173891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.173906 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:55.174039 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.174046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.174741 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.174744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.175811 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.175819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.176435 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.176438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.176590 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.176609 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.176670 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.176775 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
I20260928 21:16:55.176928 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.173229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:35579 (local address 127.5.18.194:43071)
0928 21:16:55.173471 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.173485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.173602 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.173967 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.173970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.173986 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:55.174102 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.174108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.174786 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.174789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.175822 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.175830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.176437 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.176440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.176586 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.176605 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.176668 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.176788 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":96,"thread_start_us":95,"threads_started":1}
I20260928 21:16:55.177464 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1deafa1869cd4d7c857343e09a11a176" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" is_pre_election: true
I20260928 21:16:55.177453 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1deafa1869cd4d7c857343e09a11a176" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0" is_pre_election: true
I20260928 21:16:55.177618 7363 raft_consensus.cc:2454] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:55.177691 7083 raft_consensus.cc:2454] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:55.177937 7155 leader_election.cc:302] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, 9a196c427eb34630a1954d1aefc5c137; no voters:
I20260928 21:16:55.178200 7438 raft_consensus.cc:2781] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:55.178256 7438 raft_consensus.cc:482] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:55.178297 7438 raft_consensus.cc:3037] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.179184 7438 raft_consensus.cc:504] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.179453 7438 leader_election.cc:288] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Requested vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:55.179724 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1deafa1869cd4d7c857343e09a11a176" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0"
I20260928 21:16:55.179746 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1deafa1869cd4d7c857343e09a11a176" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
I20260928 21:16:55.179836 7363 raft_consensus.cc:3037] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.179837 7083 raft_consensus.cc:3037] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.180647 7363 raft_consensus.cc:2454] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:55.180718 7083 raft_consensus.cc:2454] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:55.180891 7155 leader_election.cc:302] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, 9a196c427eb34630a1954d1aefc5c137; no voters:
I20260928 21:16:55.181037 7438 raft_consensus.cc:2781] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:55.181207 7438 raft_consensus.cc:686] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [term 1 LEADER]: Becoming Leader. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Running, Role: LEADER
I20260928 21:16:55.181700 7438 consensus_queue.cc:237] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.183631 7456 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: TxnStatusTablet state changed. Reason: New leader 9a196c427eb34630a1954d1aefc5c137. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.183734 7456 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:55.183983 7458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:16:55.184165 7458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:16:55.184509 6925 catalog_manager.cc:5495] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:55.199784 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.196316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.5.18.194:38543)
0928 21:16:55.196423 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.196428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.196533 (+ 105us) server_negotiation.cc:407] Connection header received
0928 21:16:55.196706 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.196708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.196768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.196863 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.197061 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.197645 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.198462 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.198681 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.198768 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.199045 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.199427 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.199522 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.199601 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.199648 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:16:55.199784 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.196240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.0.0.1:53596)
0928 21:16:55.196465 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.196479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.196584 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.196868 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.196871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.196878 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:55.196973 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.196979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.197735 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.197739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.198362 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.198370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.198769 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.198772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.198859 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.198907 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.199516 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.199531 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.199592 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.199672 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 21:16:55.204648 7083 raft_consensus.cc:1261] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.204650 7363 raft_consensus.cc:1261] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.205214 7456 consensus_queue.cc:1035] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:55.205389 7438 consensus_queue.cc:1035] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:55.208242 7439 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: TxnStatusTablet state changed. Reason: New leader 9a196c427eb34630a1954d1aefc5c137. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.208356 7439 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.208992 7440 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: TxnStatusTablet state changed. Reason: New leader 9a196c427eb34630a1954d1aefc5c137. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.209098 7440 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.211434 7461 mvcc.cc:205] Tried to move back new op lower bound from 7334421361472200704 to 7334421361386610688. Current Snapshot: MvccSnapshot[applied={T|T < 7334421361472200704}]
I20260928 21:16:55.211539 7438 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.211634 7438 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:55.212275 7456 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.212373 7456 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 9a196c427eb34630a1954d1aefc5c137: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:16:55.212577 7440 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.212667 7440 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P 07c1ef3ef89d4db4a4453a757b1e75e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.213472 7439 tablet_replica.cc:439] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } } }
I20260928 21:16:55.213598 7439 tablet_replica.cc:442] T 1deafa1869cd4d7c857343e09a11a176 P abbb916594374cacb8890b4478b070e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:16:55.214221 7463 mvcc.cc:205] Tried to move back new op lower bound from 7334421361472200704 to 7334421361386610688. Current Snapshot: MvccSnapshot[applied={T|T < 7334421361472200704}]
I20260928 21:16:55.215744 7462 mvcc.cc:205] Tried to move back new op lower bound from 7334421361472200704 to 7334421361386610688. Current Snapshot: MvccSnapshot[applied={T|T < 7334421361472200704}]
I20260928 21:16:55.221745 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.218098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.5.18.193:35579)
0928 21:16:55.218207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.218212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.218227 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:16:55.218365 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.218368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.218428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.218534 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.218718 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.219522 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.220303 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.220650 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.220754 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.220933 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.221396 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.221500 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.221529 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.221610 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:55.232028 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.228688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:55851 (local address 127.5.18.194:38543)
0928 21:16:55.228797 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.228800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.229021 (+ 221us) server_negotiation.cc:407] Connection header received
0928 21:16:55.229137 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.229139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.229179 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.229239 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.229464 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.229977 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.231135 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.231326 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.231389 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.231677 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.231741 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.231786 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.231889 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.231930 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:16:55.232097 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.228600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.5.18.193:55851)
0928 21:16:55.228950 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.228963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.229063 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.229256 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.229259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.229265 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:55.229379 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.229384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.230060 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.230063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.231040 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.231050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.231410 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.231414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.231504 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.231548 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.231815 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.231820 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.231872 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.231942 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260928 21:16:55.238336 7363 raft_consensus.cc:1261] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Refusing update from remote peer abbb916594374cacb8890b4478b070e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.238557 7223 raft_consensus.cc:1261] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Refusing update from remote peer abbb916594374cacb8890b4478b070e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.238698 7439 consensus_queue.cc:1035] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:55.239022 7439 consensus_queue.cc:1035] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:55.258916 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.255556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:55977 (local address 127.5.18.193:35579)
0928 21:16:55.255803 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.255807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.255850 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:16:55.255950 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.255953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.256004 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.256080 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.256279 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.256774 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.257886 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.258165 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.258268 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.258520 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.258588 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.258643 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.258739 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.258787 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260928 21:16:55.258929 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.255514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:35579 (local address 127.5.18.194:55977)
0928 21:16:55.255788 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.255801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.255885 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.256081 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.256083 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.256088 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:55.256185 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.256189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.256868 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.256872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.257780 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.257789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.258276 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.258279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.258366 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.258407 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.258656 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.258662 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.258713 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.258787 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260928 21:16:55.280622 6925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:55.280798 6925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:55.283138 6925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Sending DeleteTablet for 3 replicas of tablet 755e83ae26aa49149d1607531a12c87d
I20260928 21:16:55.283974 7343 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e83ae26aa49149d1607531a12c87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:55 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49460
I20260928 21:16:55.284397 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e83ae26aa49149d1607531a12c87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:55 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58162
I20260928 21:16:55.284792 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 755e83ae26aa49149d1607531a12c87d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:55 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53584
I20260928 21:16:55.284798 6925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:55.285183 6925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:55.285269 7483 tablet_replica.cc:330] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: stopping tablet replica
I20260928 21:16:55.285545 7483 raft_consensus.cc:2229] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:55.285706 7483 raft_consensus.cc:2258] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:55.285945 7481 tablet_replica.cc:330] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: stopping tablet replica
I20260928 21:16:55.286993 7482 tablet_replica.cc:330] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: stopping tablet replica
I20260928 21:16:55.287289 7482 raft_consensus.cc:2229] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:55.289265 7483 ts_tablet_manager.cc:1905] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:55.290504 7343 tablet_service.cc:1467] Processing CreateTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d (DEFAULT_TABLE table=test-table [id=f740c298c8c94c18a7b253f2b2776083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:55.290769 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ff4861d5dd45d8ad5295a75c42715d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.292676 7483 ts_tablet_manager.cc:1918] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:55.292831 7483 log.cc:1196] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/755e83ae26aa49149d1607531a12c87d
I20260928 21:16:55.293016 7433 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Bootstrap starting.
I20260928 21:16:55.293231 7483 ts_tablet_manager.cc:1939] T 755e83ae26aa49149d1607531a12c87d P 9a196c427eb34630a1954d1aefc5c137: Deleting consensus metadata
I20260928 21:16:55.293927 7433 tablet_bootstrap.cc:654] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.293885 7063 tablet_service.cc:1467] Processing CreateTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d (DEFAULT_TABLE table=test-table [id=f740c298c8c94c18a7b253f2b2776083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:55.294111 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ff4861d5dd45d8ad5295a75c42715d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.294678 7190 tablet_service.cc:1467] Processing CreateTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d (DEFAULT_TABLE table=test-table [id=f740c298c8c94c18a7b253f2b2776083]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:55.294889 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ff4861d5dd45d8ad5295a75c42715d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:55.296561 7432 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Bootstrap starting.
I20260928 21:16:55.290776 7481 raft_consensus.cc:2229] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:55.297454 7432 tablet_bootstrap.cc:654] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.297706 7481 raft_consensus.cc:2258] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:55.297470 7482 raft_consensus.cc:2258] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:55.298091 7434 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Bootstrap starting.
I20260928 21:16:55.298781 7433 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: No bootstrap required, opened a new log
I20260928 21:16:55.298866 7433 ts_tablet_manager.cc:1397] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 21:16:55.298919 7434 tablet_bootstrap.cc:654] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:55.298959 6913 catalog_manager.cc:4841] TS 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543): tablet 755e83ae26aa49149d1607531a12c87d (table test-table [id=5b4238421db349cba3213a3003f4cb0b]) successfully deleted
I20260928 21:16:55.299309 7433 raft_consensus.cc:348] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.299434 7433 raft_consensus.cc:374] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.299469 7433 raft_consensus.cc:729] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.299592 7433 consensus_queue.cc:260] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.299858 7433 ts_tablet_manager.cc:1428] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:16:55.300627 7481 ts_tablet_manager.cc:1905] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:55.300836 7482 ts_tablet_manager.cc:1905] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:55.301981 7432 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: No bootstrap required, opened a new log
I20260928 21:16:55.302062 7432 ts_tablet_manager.cc:1397] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260928 21:16:55.302464 7432 raft_consensus.cc:348] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.302574 7432 raft_consensus.cc:374] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.302603 7432 raft_consensus.cc:729] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb916594374cacb8890b4478b070e0, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.302723 7432 consensus_queue.cc:260] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.302963 7432 ts_tablet_manager.cc:1428] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:16:55.305074 7481 ts_tablet_manager.cc:1918] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:55.305179 7481 log.cc:1196] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/755e83ae26aa49149d1607531a12c87d
I20260928 21:16:55.305239 7482 ts_tablet_manager.cc:1918] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:55.305330 7482 log.cc:1196] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/755e83ae26aa49149d1607531a12c87d
I20260928 21:16:55.305637 7482 ts_tablet_manager.cc:1939] T 755e83ae26aa49149d1607531a12c87d P abbb916594374cacb8890b4478b070e0: Deleting consensus metadata
I20260928 21:16:55.305676 7481 ts_tablet_manager.cc:1939] T 755e83ae26aa49149d1607531a12c87d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting consensus metadata
I20260928 21:16:55.306655 6913 catalog_manager.cc:4841] TS abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579): tablet 755e83ae26aa49149d1607531a12c87d (table test-table [id=5b4238421db349cba3213a3003f4cb0b]) successfully deleted
I20260928 21:16:55.306716 6911 catalog_manager.cc:4841] TS 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355): tablet 755e83ae26aa49149d1607531a12c87d (table test-table [id=5b4238421db349cba3213a3003f4cb0b]) successfully deleted
I20260928 21:16:55.307547 7434 tablet_bootstrap.cc:492] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: No bootstrap required, opened a new log
I20260928 21:16:55.307641 7434 ts_tablet_manager.cc:1397] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260928 21:16:55.308051 7434 raft_consensus.cc:348] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.308195 7434 raft_consensus.cc:374] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:55.308225 7434 raft_consensus.cc:729] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Initialized, Role: FOLLOWER
I20260928 21:16:55.308342 7434 consensus_queue.cc:260] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.308591 7434 ts_tablet_manager.cc:1428] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:55.355643 7479 raft_consensus.cc:482] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:55.355782 7479 raft_consensus.cc:504] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.356222 7479 leader_election.cc:288] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:55.356611 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ff4861d5dd45d8ad5295a75c42715d" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0" is_pre_election: true
I20260928 21:16:55.356756 7083 raft_consensus.cc:2454] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:55.356712 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ff4861d5dd45d8ad5295a75c42715d" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" is_pre_election: true
I20260928 21:16:55.356854 7363 raft_consensus.cc:2454] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:55.357163 7155 leader_election.cc:302] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, 9a196c427eb34630a1954d1aefc5c137; no voters:
I20260928 21:16:55.357370 7479 raft_consensus.cc:2781] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:55.357424 7479 raft_consensus.cc:482] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:55.357465 7479 raft_consensus.cc:3037] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.358664 7479 raft_consensus.cc:504] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.359014 7479 leader_election.cc:288] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Requested vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:55.359400 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ff4861d5dd45d8ad5295a75c42715d" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0"
I20260928 21:16:55.359508 7083 raft_consensus.cc:3037] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.360517 7083 raft_consensus.cc:2454] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:55.360805 7157 leader_election.cc:302] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a196c427eb34630a1954d1aefc5c137, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:55.361037 7479 raft_consensus.cc:2781] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:55.361078 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ff4861d5dd45d8ad5295a75c42715d" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
I20260928 21:16:55.361138 7479 raft_consensus.cc:686] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 1 LEADER]: Becoming Leader. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Running, Role: LEADER
I20260928 21:16:55.361184 7363 raft_consensus.cc:3037] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:55.361279 7479 consensus_queue.cc:237] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:55.362193 7363 raft_consensus.cc:2454] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:55.363162 6925 catalog_manager.cc:5495] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:55.387014 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.383533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.5.18.194:38543)
0928 21:16:55.383820 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.383824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.383836 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:55.383873 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.383876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.383920 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.383983 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.384212 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.384992 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.385981 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.386310 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.386404 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.386613 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.386692 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.386758 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.386830 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.386880 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260928 21:16:55.395450 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.392430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.5.18.194:34081)
0928 21:16:55.392538 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:55.392551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:55.392612 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:55.392803 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:55.392805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:55.392810 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:55.392901 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.392904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.393812 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.393814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.394365 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:55.394372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.394892 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.394894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:55.394947 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.394969 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.395220 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.395223 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:16:55.395260 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 21:16:55.395319 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260928 21:16:55.395478 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:55.392482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:34081 (local address 127.5.18.194:38543)
0928 21:16:55.392590 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:55.392593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:55.392607 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:55.392676 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:55.392679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:55.392726 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:55.392801 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:55.392975 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.393722 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.394474 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:55.394795 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:55.394897 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:55.395063 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:55.395147 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:55.395224 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:55.395288 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:16:55.395334 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":46}
I20260928 21:16:55.400888 7363 raft_consensus.cc:1261] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.401059 7082 raft_consensus.cc:1261] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:55.401299 7438 consensus_queue.cc:1035] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:55.401810 7438 consensus_queue.cc:1035] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:16:55.413045 7461 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:16:56.964259 6925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:56.964402 6925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:56.966480 6925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Sending DeleteTablet for 3 replicas of tablet b0ff4861d5dd45d8ad5295a75c42715d
I20260928 21:16:56.966971 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:56 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58162
I20260928 21:16:56.967012 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:56 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53584
I20260928 21:16:56.967288 7514 tablet_replica.cc:330] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: stopping tablet replica
I20260928 21:16:56.967265 7343 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ff4861d5dd45d8ad5295a75c42715d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:56 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49460
I20260928 21:16:56.967397 7514 raft_consensus.cc:2229] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:56.967504 7514 raft_consensus.cc:2258] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:56.967710 7513 tablet_replica.cc:330] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: stopping tablet replica
I20260928 21:16:56.967804 7513 raft_consensus.cc:2229] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:56.968045 7513 raft_consensus.cc:2258] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:56.968464 7514 ts_tablet_manager.cc:1905] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:56.969661 7515 tablet_replica.cc:330] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: stopping tablet replica
I20260928 21:16:56.969785 7515 raft_consensus.cc:2229] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:56.969895 7515 raft_consensus.cc:2258] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:56.970453 7515 ts_tablet_manager.cc:1905] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:56.971170 7514 ts_tablet_manager.cc:1918] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:16:56.971273 7514 log.cc:1196] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/b0ff4861d5dd45d8ad5295a75c42715d
I20260928 21:16:56.971520 7514 ts_tablet_manager.cc:1939] T b0ff4861d5dd45d8ad5295a75c42715d P abbb916594374cacb8890b4478b070e0: Deleting consensus metadata
I20260928 21:16:56.972028 7513 ts_tablet_manager.cc:1905] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:56.972215 6913 catalog_manager.cc:4841] TS abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579): tablet b0ff4861d5dd45d8ad5295a75c42715d (table test-table [id=f740c298c8c94c18a7b253f2b2776083]) successfully deleted
I20260928 21:16:56.972819 7515 ts_tablet_manager.cc:1918] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:16:56.972892 7515 log.cc:1196] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/b0ff4861d5dd45d8ad5295a75c42715d
I20260928 21:16:56.973122 7515 ts_tablet_manager.cc:1939] T b0ff4861d5dd45d8ad5295a75c42715d P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting consensus metadata
I20260928 21:16:56.973773 6911 catalog_manager.cc:4841] TS 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355): tablet b0ff4861d5dd45d8ad5295a75c42715d (table test-table [id=f740c298c8c94c18a7b253f2b2776083]) successfully deleted
I20260928 21:16:56.974300 7513 ts_tablet_manager.cc:1918] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:16:56.974364 7513 log.cc:1196] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/b0ff4861d5dd45d8ad5295a75c42715d
I20260928 21:16:56.974573 7513 ts_tablet_manager.cc:1939] T b0ff4861d5dd45d8ad5295a75c42715d P 9a196c427eb34630a1954d1aefc5c137: Deleting consensus metadata
I20260928 21:16:56.975137 6913 catalog_manager.cc:4841] TS 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543): tablet b0ff4861d5dd45d8ad5295a75c42715d (table test-table [id=f740c298c8c94c18a7b253f2b2776083]) successfully deleted
Sep 28 21:16:56 dist-test-slave-rc9f krb5kdc[6881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:16:56 dist-test-slave-rc9f krb5kdc[6881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:16:56.999779 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:56.988420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.5.18.254:36773)
0928 21:16:56.988665 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:56.988675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:56.988729 (+ 54us) server_negotiation.cc:407] Connection header received
0928 21:16:56.988863 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:56.988867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:56.988913 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:56.988986 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:56.989653 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:56.990205 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:56.991009 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:56.991240 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:56.991326 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:56.996810 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:56.996813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:56.996815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:56.996842 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:56.998185 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:56.998611 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:56.998614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:56.998615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:56.998692 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:56.998935 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:56.998938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:56.998940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:56.999116 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:56.999234 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:56.999478 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 21:16:56.999614 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260928 21:16:57.002105 6925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:57.002516 6925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:57.006829 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c (DEFAULT_TABLE table=test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.006835 7063 tablet_service.cc:1467] Processing CreateTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c (DEFAULT_TABLE table=test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.006953 7343 tablet_service.cc:1467] Processing CreateTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c (DEFAULT_TABLE table=test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.007119 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efb48c9233c4a7585d139ff6c4b453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.007120 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efb48c9233c4a7585d139ff6c4b453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.007236 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efb48c9233c4a7585d139ff6c4b453c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.009615 7526 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Bootstrap starting.
I20260928 21:16:57.009974 7524 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Bootstrap starting.
I20260928 21:16:57.010561 7526 tablet_bootstrap.cc:654] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.011201 7525 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Bootstrap starting.
I20260928 21:16:57.011228 7524 tablet_bootstrap.cc:654] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.012019 7525 tablet_bootstrap.cc:654] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.012351 7524 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: No bootstrap required, opened a new log
I20260928 21:16:57.012430 7524 ts_tablet_manager.cc:1397] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:57.012887 7526 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: No bootstrap required, opened a new log
I20260928 21:16:57.012848 7524 raft_consensus.cc:348] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.012936 7524 raft_consensus.cc:374] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.012964 7524 raft_consensus.cc:729] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.012976 7526 ts_tablet_manager.cc:1397] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:57.013105 7524 consensus_queue.cc:260] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.013281 7525 tablet_bootstrap.cc:492] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: No bootstrap required, opened a new log
I20260928 21:16:57.013370 7525 ts_tablet_manager.cc:1397] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:57.013371 7526 raft_consensus.cc:348] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.013455 7524 ts_tablet_manager.cc:1428] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:57.013486 7526 raft_consensus.cc:374] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.013520 7526 raft_consensus.cc:729] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.013685 7526 consensus_queue.cc:260] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.013797 7525 raft_consensus.cc:348] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.013885 7525 raft_consensus.cc:374] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.013914 7525 raft_consensus.cc:729] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb916594374cacb8890b4478b070e0, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.013949 7526 ts_tablet_manager.cc:1428] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:57.014022 7525 consensus_queue.cc:260] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.014344 7525 ts_tablet_manager.cc:1428] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:57.176326 7508 raft_consensus.cc:482] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:57.176474 7508 raft_consensus.cc:504] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.176898 7508 leader_election.cc:288] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:57.177331 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efb48c9233c4a7585d139ff6c4b453c" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0" is_pre_election: true
I20260928 21:16:57.177349 7362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efb48c9233c4a7585d139ff6c4b453c" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" is_pre_election: true
I20260928 21:16:57.177467 7083 raft_consensus.cc:2454] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:57.177500 7362 raft_consensus.cc:2454] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:57.177810 7157 leader_election.cc:302] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a196c427eb34630a1954d1aefc5c137, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:57.177975 7508 raft_consensus.cc:2781] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:57.178032 7508 raft_consensus.cc:482] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:57.178073 7508 raft_consensus.cc:3037] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.179005 7508 raft_consensus.cc:504] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.179294 7508 leader_election.cc:288] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Requested vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:57.179615 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efb48c9233c4a7585d139ff6c4b453c" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0"
I20260928 21:16:57.179642 7362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efb48c9233c4a7585d139ff6c4b453c" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
I20260928 21:16:57.179737 7362 raft_consensus.cc:3037] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.179759 7083 raft_consensus.cc:3037] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.180603 7362 raft_consensus.cc:2454] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:57.180605 7083 raft_consensus.cc:2454] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 1.
I20260928 21:16:57.180850 7155 leader_election.cc:302] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, 9a196c427eb34630a1954d1aefc5c137; no voters:
I20260928 21:16:57.181005 7508 raft_consensus.cc:2781] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:57.181092 7508 raft_consensus.cc:686] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 1 LEADER]: Becoming Leader. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Running, Role: LEADER
I20260928 21:16:57.181227 7508 consensus_queue.cc:237] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.182945 6924 catalog_manager.cc:5495] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "9a196c427eb34630a1954d1aefc5c137" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:57.204131 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.201168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.5.18.194:38543)
0928 21:16:57.201426 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.201432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.201442 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:16:57.201474 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.201476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.201513 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.201603 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:57.201834 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.202344 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.203141 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.203363 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.203446 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.203680 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.203770 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.203826 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.203930 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.203980 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260928 21:16:57.209141 7362 raft_consensus.cc:1261] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:57.209285 7083 raft_consensus.cc:1261] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Refusing update from remote peer 9a196c427eb34630a1954d1aefc5c137: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:57.209525 7491 consensus_queue.cc:1035] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:57.209681 7494 consensus_queue.cc:1035] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:57.238117 6925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:57.238293 6925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:57.240013 6925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Sending DeleteTablet for 3 replicas of tablet 7efb48c9233c4a7585d139ff6c4b453c
I20260928 21:16:57.240439 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:57 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58162
I20260928 21:16:57.240566 7343 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:57 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49460
I20260928 21:16:57.240636 7514 tablet_replica.cc:330] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: stopping tablet replica
I20260928 21:16:57.240749 7514 raft_consensus.cc:2229] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:57.240761 7515 tablet_replica.cc:330] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: stopping tablet replica
I20260928 21:16:57.240840 7514 raft_consensus.cc:2258] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:57.240867 7515 raft_consensus.cc:2229] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:57.240865 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efb48c9233c4a7585d139ff6c4b453c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:57 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53584
I20260928 21:16:57.240965 7515 raft_consensus.cc:2258] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:57.241006 7513 tablet_replica.cc:330] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: stopping tablet replica
I20260928 21:16:57.241091 7513 raft_consensus.cc:2229] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:57.241155 6925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:16:57.241248 7513 raft_consensus.cc:2258] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:16:57.241523 6925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:57.241673 7515 ts_tablet_manager.cc:1905] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:57.241803 7514 ts_tablet_manager.cc:1905] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:57.241834 7513 ts_tablet_manager.cc:1905] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:57.243909 7515 ts_tablet_manager.cc:1918] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:57.243961 7515 log.cc:1196] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/7efb48c9233c4a7585d139ff6c4b453c
I20260928 21:16:57.244065 7513 ts_tablet_manager.cc:1918] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:57.244065 7514 ts_tablet_manager.cc:1918] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:16:57.244132 7515 ts_tablet_manager.cc:1939] T 7efb48c9233c4a7585d139ff6c4b453c P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting consensus metadata
I20260928 21:16:57.244127 7514 log.cc:1196] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/7efb48c9233c4a7585d139ff6c4b453c
I20260928 21:16:57.244127 7513 log.cc:1196] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/7efb48c9233c4a7585d139ff6c4b453c
I20260928 21:16:57.244355 7514 ts_tablet_manager.cc:1939] T 7efb48c9233c4a7585d139ff6c4b453c P abbb916594374cacb8890b4478b070e0: Deleting consensus metadata
I20260928 21:16:57.244355 7513 ts_tablet_manager.cc:1939] T 7efb48c9233c4a7585d139ff6c4b453c P 9a196c427eb34630a1954d1aefc5c137: Deleting consensus metadata
I20260928 21:16:57.244681 6911 catalog_manager.cc:4841] TS 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355): tablet 7efb48c9233c4a7585d139ff6c4b453c (table test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]) successfully deleted
I20260928 21:16:57.244931 6913 catalog_manager.cc:4841] TS 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543): tablet 7efb48c9233c4a7585d139ff6c4b453c (table test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]) successfully deleted
I20260928 21:16:57.245323 6913 catalog_manager.cc:4841] TS abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579): tablet 7efb48c9233c4a7585d139ff6c4b453c (table test-table [id=1ec43e2b5f7c485aae8f1436582f3a0b]) successfully deleted
I20260928 21:16:57.247584 7190 tablet_service.cc:1467] Processing CreateTablet for tablet eb01825e3c51400480c33b25e9fbcacc (DEFAULT_TABLE table=test-table [id=7c8d5e0e31fa412199d2b36a1cbbd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.247565 7063 tablet_service.cc:1467] Processing CreateTablet for tablet eb01825e3c51400480c33b25e9fbcacc (DEFAULT_TABLE table=test-table [id=7c8d5e0e31fa412199d2b36a1cbbd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.247807 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01825e3c51400480c33b25e9fbcacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.247807 7063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01825e3c51400480c33b25e9fbcacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.249828 7525 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: Bootstrap starting.
I20260928 21:16:57.249828 7526 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Bootstrap starting.
I20260928 21:16:57.250052 7343 tablet_service.cc:1467] Processing CreateTablet for tablet eb01825e3c51400480c33b25e9fbcacc (DEFAULT_TABLE table=test-table [id=7c8d5e0e31fa412199d2b36a1cbbd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:57.250245 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb01825e3c51400480c33b25e9fbcacc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:57.250597 7526 tablet_bootstrap.cc:654] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.250597 7525 tablet_bootstrap.cc:654] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.251752 7526 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: No bootstrap required, opened a new log
I20260928 21:16:57.251857 7526 ts_tablet_manager.cc:1397] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:57.251922 7525 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: No bootstrap required, opened a new log
I20260928 21:16:57.251994 7525 ts_tablet_manager.cc:1397] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:57.252246 7524 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Bootstrap starting.
I20260928 21:16:57.252239 7526 raft_consensus.cc:348] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.252341 7526 raft_consensus.cc:374] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.252370 7526 raft_consensus.cc:729] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a196c427eb34630a1954d1aefc5c137, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.252323 7525 raft_consensus.cc:348] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.252426 7525 raft_consensus.cc:374] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.252454 7525 raft_consensus.cc:729] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb916594374cacb8890b4478b070e0, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.252485 7526 consensus_queue.cc:260] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.252565 7525 consensus_queue.cc:260] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.252727 7526 ts_tablet_manager.cc:1428] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:16:57.252811 7525 ts_tablet_manager.cc:1428] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:16:57.253118 7524 tablet_bootstrap.cc:654] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:57.254303 7524 tablet_bootstrap.cc:492] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: No bootstrap required, opened a new log
I20260928 21:16:57.254387 7524 ts_tablet_manager.cc:1397] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:16:57.254791 7524 raft_consensus.cc:348] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.254897 7524 raft_consensus.cc:374] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:57.254928 7524 raft_consensus.cc:729] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Initialized, Role: FOLLOWER
I20260928 21:16:57.255043 7524 consensus_queue.cc:260] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.255296 7524 ts_tablet_manager.cc:1428] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:16:57.259336 7530 raft_consensus.cc:482] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:57.259419 7530 raft_consensus.cc:504] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.260349 7530 leader_election.cc:288] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543)
I20260928 21:16:57.264277 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.261015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:45011 (local address 127.5.18.194:38543)
0928 21:16:57.261125 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.261129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.261255 (+ 126us) server_negotiation.cc:407] Connection header received
0928 21:16:57.261336 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.261339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.261381 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.261443 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:57.261724 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.262238 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263385 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.263801 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263896 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.263943 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.264103 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.264169 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:16:57.264285 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.260595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:49897 (local address 127.5.18.193:35579)
0928 21:16:57.260788 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.260800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.260815 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:16:57.260948 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.260951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.260998 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.261071 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:57.261418 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.262031 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263333 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.263776 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263873 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.263924 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.264080 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.264150 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260928 21:16:57.264307 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.260863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.5.18.195:45011)
0928 21:16:57.261187 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:57.261197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:57.261270 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:57.261464 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:57.261467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:57.261483 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:57.261617 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.261625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.262339 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.262342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.263279 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.263287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.263896 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.263998 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.264015 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:16:57.264081 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:16:57.264146 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":20,"thread_start_us":87,"threads_started":1}
I20260928 21:16:57.264307 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.260537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:35579 (local address 127.5.18.195:49897)
0928 21:16:57.260746 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:57.260760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:57.260848 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:57.261180 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:57.261183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:57.261201 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:57.261310 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.261317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.262176 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.262179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.263224 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.263233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.263879 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.263883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.263975 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.263997 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:16:57.264059 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:16:57.264126 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260928 21:16:57.264709 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0" is_pre_election: true
I20260928 21:16:57.264832 7083 raft_consensus.cc:2454] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in term 0.
I20260928 21:16:57.264844 7491 raft_consensus.cc:482] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:57.264844 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a196c427eb34630a1954d1aefc5c137" is_pre_election: true
I20260928 21:16:57.264940 7491 raft_consensus.cc:504] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.265255 7491 leader_election.cc:288] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:57.265358 7223 raft_consensus.cc:2454] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in term 0.
I20260928 21:16:57.265611 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" is_pre_election: true
I20260928 21:16:57.265650 7297 leader_election.cc:302] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:57.265729 7363 raft_consensus.cc:2454] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:57.265879 7530 raft_consensus.cc:2781] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:57.265936 7530 raft_consensus.cc:482] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:57.265973 7530 raft_consensus.cc:3037] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.265962 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0" is_pre_election: true
I20260928 21:16:57.265988 7155 leader_election.cc:302] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, 9a196c427eb34630a1954d1aefc5c137; no voters:
I20260928 21:16:57.266062 7083 raft_consensus.cc:2454] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a196c427eb34630a1954d1aefc5c137 in term 0.
I20260928 21:16:57.266963 7530 raft_consensus.cc:504] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.267284 7530 leader_election.cc:288] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [CANDIDATE]: Term 1 election: Requested vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543)
I20260928 21:16:57.267403 7491 raft_consensus.cc:2781] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:57.267450 7491 raft_consensus.cc:482] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:57.267486 7491 raft_consensus.cc:3037] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.267598 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0"
I20260928 21:16:57.267586 7223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a196c427eb34630a1954d1aefc5c137"
I20260928 21:16:57.267683 7083 raft_consensus.cc:3037] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:57.268356 7491 raft_consensus.cc:504] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.268496 7083 raft_consensus.cc:2454] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in term 1.
I20260928 21:16:57.268577 7223 raft_consensus.cc:2379] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in current term 1: Already voted for candidate 9a196c427eb34630a1954d1aefc5c137 in this term.
I20260928 21:16:57.268674 7491 leader_election.cc:288] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Requested vote from peers abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579), 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355)
I20260928 21:16:57.268791 7297 leader_election.cc:302] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c1ef3ef89d4db4a4453a757b1e75e5, abbb916594374cacb8890b4478b070e0; no voters:
I20260928 21:16:57.269049 7083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbb916594374cacb8890b4478b070e0"
I20260928 21:16:57.269189 7083 raft_consensus.cc:2379] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a196c427eb34630a1954d1aefc5c137 in current term 1: Already voted for candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in this term.
I20260928 21:16:57.269164 7363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb01825e3c51400480c33b25e9fbcacc" candidate_uuid: "9a196c427eb34630a1954d1aefc5c137" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5"
I20260928 21:16:57.269294 7363 raft_consensus.cc:2379] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9a196c427eb34630a1954d1aefc5c137 in current term 1: Already voted for candidate 07c1ef3ef89d4db4a4453a757b1e75e5 in this term.
I20260928 21:16:57.269443 7530 raft_consensus.cc:2781] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:57.269511 7155 leader_election.cc:302] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9a196c427eb34630a1954d1aefc5c137; no voters: 07c1ef3ef89d4db4a4453a757b1e75e5, abbb916594374cacb8890b4478b070e0
I20260928 21:16:57.269649 7530 raft_consensus.cc:686] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 LEADER]: Becoming Leader. State: Replica: 07c1ef3ef89d4db4a4453a757b1e75e5, State: Running, Role: LEADER
I20260928 21:16:57.269662 7491 raft_consensus.cc:2726] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 21:16:57.269870 7530 consensus_queue.cc:237] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } }
I20260928 21:16:57.271667 6925 catalog_manager.cc:5495] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c1ef3ef89d4db4a4453a757b1e75e5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37355 } health_report { overall_health: HEALTHY } } }
I20260928 21:16:57.290689 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.287172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.5.18.195:37355)
0928 21:16:57.287470 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.287478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.287490 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:57.287558 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.287560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.287609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.287715 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:57.287936 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.288474 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.289264 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.289488 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.289595 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.289857 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.290312 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.290414 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.290478 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.290532 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":137,"threads_started":1}
I20260928 21:16:57.299815 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.296892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46393 (local address 127.5.18.194:38543)
0928 21:16:57.296998 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.297002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.297171 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:16:57.297260 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.297262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.297305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.297362 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:57.297634 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.298121 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.298937 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.299180 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.299253 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.299492 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.299568 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.299609 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.299669 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.299711 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:16:57.299839 7549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.296811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38543 (local address 127.5.18.195:46393)
0928 21:16:57.297098 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:57.297111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:57.297196 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:57.297376 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:57.297378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:57.297383 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:57.297477 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.297482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.298227 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.298236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.298843 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.298850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.299278 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.299281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.299347 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.299386 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.299613 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.299618 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:16:57.299662 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:16:57.299721 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260928 21:16:57.304868 7083 raft_consensus.cc:1261] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0 [term 1 FOLLOWER]: Refusing update from remote peer 07c1ef3ef89d4db4a4453a757b1e75e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:57.304908 7223 raft_consensus.cc:1261] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Refusing update from remote peer 07c1ef3ef89d4db4a4453a757b1e75e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:57.305270 7544 consensus_queue.cc:1035] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbb916594374cacb8890b4478b070e0" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:57.305442 7530 consensus_queue.cc:1035] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a196c427eb34630a1954d1aefc5c137" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:57.322026 7546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.317508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45855 (local address 127.5.18.195:37355)
0928 21:16:57.317627 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:57.317632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:57.317792 (+ 160us) server_negotiation.cc:407] Connection header received
0928 21:16:57.317929 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:57.317932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:57.317979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:57.318045 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:57.318261 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.318752 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.319758 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.320033 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.320122 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.321654 (+ 1532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.321724 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.321783 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:57.321890 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:16:57.321935 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:16:57.322091 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:57.317457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37355 (local address 127.5.18.194:45855)
0928 21:16:57.317719 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:57.317733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:57.317831 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:57.318068 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:57.318071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:57.318079 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:16:57.318174 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.318179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.318848 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.318851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.319439 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:57.319446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:57.320206 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:57.320209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:57.320293 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:57.320322 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:16:57.321806 (+ 1484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:16:57.321812 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:16:57.321873 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:16:57.321954 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":123,"threads_started":1}
W20260928 21:16:57.322693 7495 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:16:58.873075 6925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:58.873219 6925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:58.875490 6925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf4be8b432c34d41ac5e758723cdc350: Sending DeleteTablet for 3 replicas of tablet eb01825e3c51400480c33b25e9fbcacc
I20260928 21:16:58.876108 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01825e3c51400480c33b25e9fbcacc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:58 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53584
I20260928 21:16:58.876099 7063 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01825e3c51400480c33b25e9fbcacc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:58 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58162
I20260928 21:16:58.876164 7343 tablet_service.cc:1514] Processing DeleteTablet for tablet eb01825e3c51400480c33b25e9fbcacc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:58 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49460
I20260928 21:16:58.876475 7580 tablet_replica.cc:330] T eb01825e3c51400480c33b25e9fbcacc P abbb916594374cacb8890b4478b070e0: stopping tablet replica
I20260928 21:16:58.876493 7579 tablet_replica.cc:330] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: stopping tablet replica
I20260928 21:16:58.876528 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6998
I20260928 21:16:58.877424 7581 tablet_replica.cc:330] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: stopping tablet replica
I20260928 21:16:58.877727 7581 raft_consensus.cc:2229] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:58.877893 7579 raft_consensus.cc:2229] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:58.878096 7579 raft_consensus.cc:2258] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:58.878237 7581 raft_consensus.cc:2258] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:58.878806 7579 ts_tablet_manager.cc:1905] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:58.878870 7581 ts_tablet_manager.cc:1905] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:58.881068 7581 ts_tablet_manager.cc:1918] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:16:58.881141 7581 log.cc:1196] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/eb01825e3c51400480c33b25e9fbcacc
I20260928 21:16:58.881371 7579 ts_tablet_manager.cc:1918] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:16:58.881422 7581 ts_tablet_manager.cc:1939] T eb01825e3c51400480c33b25e9fbcacc P 07c1ef3ef89d4db4a4453a757b1e75e5: Deleting consensus metadata
I20260928 21:16:58.881443 7579 log.cc:1196] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/eb01825e3c51400480c33b25e9fbcacc
I20260928 21:16:58.881693 7579 ts_tablet_manager.cc:1939] T eb01825e3c51400480c33b25e9fbcacc P 9a196c427eb34630a1954d1aefc5c137: Deleting consensus metadata
I20260928 21:16:58.882126 6911 catalog_manager.cc:4841] TS 07c1ef3ef89d4db4a4453a757b1e75e5 (127.5.18.195:37355): tablet eb01825e3c51400480c33b25e9fbcacc (table test-table [id=7c8d5e0e31fa412199d2b36a1cbbd8dc]) successfully deleted
I20260928 21:16:58.882306 6913 catalog_manager.cc:4841] TS 9a196c427eb34630a1954d1aefc5c137 (127.5.18.194:38543): tablet eb01825e3c51400480c33b25e9fbcacc (table test-table [id=7c8d5e0e31fa412199d2b36a1cbbd8dc]) successfully deleted
W20260928 21:16:58.884268 6913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:35579 (error 108)
I20260928 21:16:58.884435 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7138
I20260928 21:16:58.885833 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:58.885417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:35579 (local address 127.0.0.1:58174)
0928 21:16:58.885663 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:58.885739 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:35579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
W20260928 21:16:58.885972 6913 catalog_manager.cc:4584] TS abbb916594374cacb8890b4478b070e0 (127.5.18.193:35579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb01825e3c51400480c33b25e9fbcacc: Network error: Client connection negotiation failed: client connection to 127.5.18.193:35579: connect: Connection refused (error 111)
I20260928 21:16:58.891134 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7278
I20260928 21:16:58.896392 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 6894
2026-09-28T21:16:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:58 dist-test-slave-rc9f krb5kdc[7586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:58 dist-test-slave-rc9f krb5kdc[7586](info): set up 2 sockets
Sep 28 21:16:58 dist-test-slave-rc9f krb5kdc[7586](info): commencing operation
krb5kdc: starting...
W20260928 21:16:59.849197 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:16:59 dist-test-slave-rc9f krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:16:59.960748 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:39805
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:00.065730 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:00.066015 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:00.066071 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:00.070322 7599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:00.070397 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:00.070426 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:00.070448 7599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:00.070479 7599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:00.076133 7599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:00.077204 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:00.078248 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:00.086661 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.086664 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.086634 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:00.086640 7599 server_base.cc:1034] running on GCE node
I20260928 21:17:00.087401 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:00.088430 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:00.089612 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790630220089594 us; error 37 us; skew 500 ppm
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:00.092137 7599 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:00.092950 7599 webserver.cc:460] Webserver started at http://127.5.18.254:45825/ using document root <none> and password file <none>
I20260928 21:17:00.093463 7599 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:00.093524 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:00.093729 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:00.095662 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "1ca12ab2819943939e3f363bf033ed9a"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "691b8bb09eb98fe69edbf2c6afaa5272"
server_key_iv: "767de2d1e78d9c766e9d65e78e73625b"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.096177 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "1ca12ab2819943939e3f363bf033ed9a"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "691b8bb09eb98fe69edbf2c6afaa5272"
server_key_iv: "767de2d1e78d9c766e9d65e78e73625b"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.099745 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:00.102080 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.103255 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:00.103376 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "1ca12ab2819943939e3f363bf033ed9a"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "691b8bb09eb98fe69edbf2c6afaa5272"
server_key_iv: "767de2d1e78d9c766e9d65e78e73625b"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.103467 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:00.116858 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:00.119614 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:00.119776 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:00.126319 7599 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:39805
I20260928 21:17:00.126322 7666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:39805 every 8 connection(s)
I20260928 21:17:00.127455 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:00.130158 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:00.135458 7667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a: Bootstrap starting.
I20260928 21:17:00.136184 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7599
I20260928 21:17:00.136281 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:00.136508 5195 external_mini_cluster.cc:1428] Setting key 4331a19ab493a5ccb4f1d8ec85807858
I20260928 21:17:00.138118 7667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:00.138979 7667 log.cc:824] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:00.140978 7667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a: No bootstrap required, opened a new log
I20260928 21:17:00.144847 7667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } }
I20260928 21:17:00.145112 7667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:00.145169 7667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca12ab2819943939e3f363bf033ed9a, State: Initialized, Role: FOLLOWER
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:00.145642 7667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } }
I20260928 21:17:00.145779 7667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:00.145840 7667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:00.145927 7667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:00.146874 7667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } }
I20260928 21:17:00.147195 7667 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ca12ab2819943939e3f363bf033ed9a; no voters:
I20260928 21:17:00.147461 7667 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:00.147554 7672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:00.147753 7672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [term 1 LEADER]: Becoming Leader. State: Replica: 1ca12ab2819943939e3f363bf033ed9a, State: Running, Role: LEADER
I20260928 21:17:00.148118 7672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } }
I20260928 21:17:00.148717 7667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:00.149868 7674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca12ab2819943939e3f363bf033ed9a. Latest consensus state: current_term: 1 leader_uuid: "1ca12ab2819943939e3f363bf033ed9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } } }
I20260928 21:17:00.149868 7673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ca12ab2819943939e3f363bf033ed9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca12ab2819943939e3f363bf033ed9a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39805 } } }
I20260928 21:17:00.150005 7673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:00.150005 7674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:00.150408 7681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:00.151569 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.137682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51016 (local address 127.5.18.254:39805)
0928 21:17:00.138140 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:00.138159 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:00.138210 (+ 51us) server_negotiation.cc:407] Connection header received
0928 21:17:00.139017 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:00.139042 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:00.139456 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:00.139778 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:00.140391 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.141823 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.142615 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.143299 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.143462 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.146015 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:00.146021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:00.146037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:00.146069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:00.147916 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.148392 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.148398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.148404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.148540 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.148842 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.148845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.148847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.149348 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:00.149535 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:00.150418 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 21:17:00.150731 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":119,"threads_started":1}
I20260928 21:17:00.153836 7681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:00.159197 7681 catalog_manager.cc:1347] Generated new cluster ID: 9b320beb93d54f449b8f920c65f102cf
I20260928 21:17:00.159250 7681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:00.177142 7681 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:00.178180 7681 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:00.185326 7681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca12ab2819943939e3f363bf033ed9a: Generated new TSK 0
I20260928 21:17:00.186003 7681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:00.222419 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:00.351817 7693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:00.352075 7693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:00.352118 7693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:00.356076 7693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:00.356139 7693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:00.356206 7693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:00.361639 7693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:00.362659 7693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:00.363652 7693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:00.372723 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.372746 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:00.372762 7693 server_base.cc:1034] running on GCE node
W20260928 21:17:00.372743 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:00.373307 7693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:00.373907 7693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:00.375054 7693 hybrid_clock.cc:648] HybridClock initialized: now 1790630220375035 us; error 30 us; skew 500 ppm
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:00.377672 7693 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:00.378469 7693 webserver.cc:460] Webserver started at http://127.5.18.193:40293/ using document root <none> and password file <none>
I20260928 21:17:00.378971 7693 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:00.379026 7693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:00.379206 7693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:00.381124 7693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "20f9da2971514ab397b1f374207178a8"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "f10e3f9cc0fd6965f1a86858559f9090"
server_key_iv: "d589896f2e87978e0cca2b7a0595eded"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.381666 7693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "20f9da2971514ab397b1f374207178a8"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "f10e3f9cc0fd6965f1a86858559f9090"
server_key_iv: "d589896f2e87978e0cca2b7a0595eded"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.385011 7693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:17:00.387229 7708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.388403 7693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:00.388501 7693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "20f9da2971514ab397b1f374207178a8"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "f10e3f9cc0fd6965f1a86858559f9090"
server_key_iv: "d589896f2e87978e0cca2b7a0595eded"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.388597 7693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:00.409823 7693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:00.412667 7693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:00.412832 7693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:00.413378 7693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:00.414166 7693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:00.414213 7693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.414263 7693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:00.414291 7693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.422116 7693 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:44627
I20260928 21:17:00.422166 7821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:44627 every 8 connection(s)
I20260928 21:17:00.422941 7693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:00.428880 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7693
I20260928 21:17:00.428992 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:00.429279 5195 external_mini_cluster.cc:1428] Setting key db2415b6ead7434fdb8242727fb5baba
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:00.435819 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.424566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:35933 (local address 127.5.18.254:39805)
0928 21:17:00.424683 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:00.424687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:00.425451 (+ 764us) server_negotiation.cc:407] Connection header received
0928 21:17:00.426730 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:00.426733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:00.426777 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:00.426846 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:00.427925 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.428439 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.429861 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.430126 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.430214 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.432662 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:00.432665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:00.432667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:00.432688 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:00.433972 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.434438 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.434442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.434443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.434516 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.434813 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.434816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.434818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.434985 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:00.435066 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:00.435539 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 21:17:00.435665 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:17:00.436575 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.424923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:39805 (local address 127.5.18.193:35933)
0928 21:17:00.425315 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:00.425353 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:00.426550 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:00.426987 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:00.426994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:00.427326 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:00.427812 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.427826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.428524 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.428526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.429761 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.429770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.430269 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.430273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.430440 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.430979 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:00.430999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:00.432498 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:00.434076 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.434082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.434096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.434348 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.434593 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.434596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.434599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.434731 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.435079 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:00.435086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:00.435320 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 21:17:00.435481 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 21:17:00.435786 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:17:00.437813 7822 heartbeater.cc:347] Connected to a master server at 127.5.18.254:39805
I20260928 21:17:00.438011 7822 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:00.438448 7822 heartbeater.cc:511] Master 127.5.18.254:39805 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:00.439896 7631 ts_manager.cc:194] Registered new tserver with Master: 20f9da2971514ab397b1f374207178a8 (127.5.18.193:44627)
I20260928 21:17:00.441021 7631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:35933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:00.454185 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:00.562618 7827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:00.562885 7827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:00.562937 7827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:00.566920 7827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:00.566989 7827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:00.567065 7827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:00.572356 7827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:00.573387 7827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:00.574438 7827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:00.583781 7835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.583847 7833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.583847 7832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:00.583989 7827 server_base.cc:1034] running on GCE node
I20260928 21:17:00.584511 7827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:00.585060 7827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:00.586212 7827 hybrid_clock.cc:648] HybridClock initialized: now 1790630220586196 us; error 27 us; skew 500 ppm
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:00.588733 7827 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:00.589522 7827 webserver.cc:460] Webserver started at http://127.5.18.194:35185/ using document root <none> and password file <none>
I20260928 21:17:00.590039 7827 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:00.590104 7827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:00.590293 7827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:00.592312 7827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "c1cd6f1a1055459bbc4e57cbd39ba12f"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "7e177db7ce6d8e50285aadbded027c5a"
server_key_iv: "72ab78f703874e127436c5245cbf9eb2"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.592830 7827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "c1cd6f1a1055459bbc4e57cbd39ba12f"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "7e177db7ce6d8e50285aadbded027c5a"
server_key_iv: "72ab78f703874e127436c5245cbf9eb2"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.596271 7827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 21:17:00.598455 7842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.599596 7827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:00.599707 7827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c1cd6f1a1055459bbc4e57cbd39ba12f"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "7e177db7ce6d8e50285aadbded027c5a"
server_key_iv: "72ab78f703874e127436c5245cbf9eb2"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.599802 7827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:00.619051 7827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:00.621773 7827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:00.621932 7827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:00.622467 7827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:00.623241 7827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:00.623284 7827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.623325 7827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:00.623353 7827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.631404 7827 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:38735
I20260928 21:17:00.631428 7955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:38735 every 8 connection(s)
I20260928 21:17:00.632251 7827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:00.639637 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7827
I20260928 21:17:00.639741 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:00.639990 5195 external_mini_cluster.cc:1428] Setting key 543d579de447a47a02708797c7285670
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:00.645502 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.633956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:35639 (local address 127.5.18.254:39805)
0928 21:17:00.634096 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:00.634100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:00.634934 (+ 834us) server_negotiation.cc:407] Connection header received
0928 21:17:00.636257 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:00.636261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:00.636319 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:00.636392 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:00.637525 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.638084 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.639189 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.639404 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.639492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.642190 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:00.642194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:00.642196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:00.642223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:00.643553 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.644068 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.644072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.644074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.644161 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.644443 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.644445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.644447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.644649 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:00.644754 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:00.645210 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 21:17:00.645330 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:17:00.646294 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.634322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:39805 (local address 127.5.18.194:35639)
0928 21:17:00.634775 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:00.634817 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:00.636048 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:00.636586 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:00.636593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:00.636927 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:00.637386 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.637399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.638201 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.638204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.639074 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.639083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.639546 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.639551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.639712 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.640318 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:00.640340 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:00.642005 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:00.643657 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.643663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.643679 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.643969 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.644245 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.644247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.644248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.644370 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.644741 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:00.644746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:00.644963 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 21:17:00.645138 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 21:17:00.645385 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":101,"threads_started":1}
I20260928 21:17:00.647441 7956 heartbeater.cc:347] Connected to a master server at 127.5.18.254:39805
I20260928 21:17:00.647672 7956 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:00.648164 7956 heartbeater.cc:511] Master 127.5.18.254:39805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:00.649201 7631 ts_manager.cc:194] Registered new tserver with Master: c1cd6f1a1055459bbc4e57cbd39ba12f (127.5.18.194:38735)
I20260928 21:17:00.649871 7631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:35639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:00.664108 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:00.769304 7961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:00.769590 7961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:00.769649 7961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:00.773676 7961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:00.773729 7961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:00.773794 7961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:00.779217 7961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:00.780294 7961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:00.781240 7961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:00.790218 7967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.790292 7969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:00.790271 7966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:00.790343 7961 server_base.cc:1034] running on GCE node
I20260928 21:17:00.790831 7961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:00.791442 7961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:00.792696 7961 hybrid_clock.cc:648] HybridClock initialized: now 1790630220792657 us; error 50 us; skew 500 ppm
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:00.795320 7961 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:00.796090 7961 webserver.cc:460] Webserver started at http://127.5.18.195:40123/ using document root <none> and password file <none>
I20260928 21:17:00.796573 7961 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:00.796625 7961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:00.796784 7961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:00.798715 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "f08def6539f741ababe51d4244332b95"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "238860d6db7027d1d7b4d8b3573254f0"
server_key_iv: "a138ab602c519af4b51752762937574e"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.799199 7961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "f08def6539f741ababe51d4244332b95"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "238860d6db7027d1d7b4d8b3573254f0"
server_key_iv: "a138ab602c519af4b51752762937574e"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.802597 7961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:00.804881 7976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.806118 7961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:00.806217 7961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "f08def6539f741ababe51d4244332b95"
format_stamp: "Formatted at 2026-09-28 21:17:00 on dist-test-slave-rc9f"
server_key: "238860d6db7027d1d7b4d8b3573254f0"
server_key_iv: "a138ab602c519af4b51752762937574e"
server_key_version: "encryptionkey@0"
I20260928 21:17:00.806308 7961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:00.826821 7961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:00.829638 7961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:00.829807 7961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:00.830428 7961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:00.831238 7961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:00.831291 7961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.831343 7961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:00.831367 7961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:00.839265 7961 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:36019
I20260928 21:17:00.839300 8089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:36019 every 8 connection(s)
I20260928 21:17:00.840085 7961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[7586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:00.849684 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 7961
I20260928 21:17:00.849771 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:00.849995 5195 external_mini_cluster.cc:1428] Setting key 09a24afcf15a0dfbfd9ef2997d187eda
I20260928 21:17:00.852993 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.841704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56983 (local address 127.5.18.254:39805)
0928 21:17:00.841832 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:00.841837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:00.842670 (+ 833us) server_negotiation.cc:407] Connection header received
0928 21:17:00.843889 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:00.843891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:00.843936 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:00.844004 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:00.845173 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.845843 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.846923 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.847167 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.847256 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.849492 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:00.849496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:00.849499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:00.849523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:00.850962 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.851570 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.851573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.851575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.851658 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:00.851993 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:00.851996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:00.851998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:00.852184 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:00.852267 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:00.852727 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 21:17:00.852840 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:00.853801 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:00.842113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:39805 (local address 127.5.18.195:56983)
0928 21:17:00.842528 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:00.842568 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:00.843716 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:00.844146 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:00.844152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:00.844471 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:00.845040 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.845057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.845952 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.845956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.846814 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:00.846822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:00.847302 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:00.847306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:00.847464 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:00.847996 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:00.848011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:00.849334 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:00.851182 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.851187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.851201 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.851478 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.851747 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:00.851749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:00.851752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:00.851920 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:00.852280 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:00.852287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:00.852512 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 21:17:00.852676 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 21:17:00.852977 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260928 21:17:00.854940 8090 heartbeater.cc:347] Connected to a master server at 127.5.18.254:39805
I20260928 21:17:00.855139 8090 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:00.855566 8090 heartbeater.cc:511] Master 127.5.18.254:39805 requested a full tablet report, sending...
I20260928 21:17:00.856463 7631 ts_manager.cc:194] Registered new tserver with Master: f08def6539f741ababe51d4244332b95 (127.5.18.195:36019)
I20260928 21:17:00.856992 7631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:56983
I20260928 21:17:00.863440 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:17:00.875228 7670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:00.873878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.5.18.254:39805)
0928 21:17:00.874015 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:00.874019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:00.874165 (+ 146us) server_negotiation.cc:407] Connection header received
0928 21:17:00.874301 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:00.874304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:00.874347 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:00.874407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:00.875111 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51032: BlockingRecv error: recv got EOF from 127.0.0.1:51032 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:17:00.876462 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7693
I20260928 21:17:00.881095 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7827
I20260928 21:17:00.885869 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7961
I20260928 21:17:00.889972 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 7599
2026-09-28T21:17:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1992 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[8102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[8102](info): set up 2 sockets
Sep 28 21:17:00 dist-test-slave-rc9f krb5kdc[8102](info): commencing operation
krb5kdc: starting...
W20260928 21:17:01.849200 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:01 dist-test-slave-rc9f krb5kdc[8102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:01.963904 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45001
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:02.065762 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:02.066013 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:02.066068 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:02.069967 8115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:02.070021 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:02.070047 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:02.070070 8115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:02.070097 8115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:02.075109 8115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:45001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:02.076035 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:02.076978 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:02.084738 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:02.084841 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:02.084853 8115 server_base.cc:1034] running on GCE node
W20260928 21:17:02.084740 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:02.085635 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:02.086707 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:02.087859 8115 hybrid_clock.cc:648] HybridClock initialized: now 1790630222087844 us; error 26 us; skew 500 ppm
Sep 28 21:17:02 dist-test-slave-rc9f krb5kdc[8102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630222, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:02.095594 8115 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:02.096457 8115 webserver.cc:460] Webserver started at http://127.5.18.254:41013/ using document root <none> and password file <none>
I20260928 21:17:02.096968 8115 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:02.097038 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:02.097229 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:02.099195 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "5d68c2ef70d94b59985f20cb315ce5b4"
format_stamp: "Formatted at 2026-09-28 21:17:02 on dist-test-slave-rc9f"
server_key: "dbff25a21e6afbae088024cb4531d9d0"
server_key_iv: "d9a83c5b8f7942c71a43d002cc3dc447"
server_key_version: "encryptionkey@0"
I20260928 21:17:02.099711 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "5d68c2ef70d94b59985f20cb315ce5b4"
format_stamp: "Formatted at 2026-09-28 21:17:02 on dist-test-slave-rc9f"
server_key: "dbff25a21e6afbae088024cb4531d9d0"
server_key_iv: "d9a83c5b8f7942c71a43d002cc3dc447"
server_key_version: "encryptionkey@0"
I20260928 21:17:02.103160 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:02.105360 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:02.106492 8115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:02.106595 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "5d68c2ef70d94b59985f20cb315ce5b4"
format_stamp: "Formatted at 2026-09-28 21:17:02 on dist-test-slave-rc9f"
server_key: "dbff25a21e6afbae088024cb4531d9d0"
server_key_iv: "d9a83c5b8f7942c71a43d002cc3dc447"
server_key_version: "encryptionkey@0"
I20260928 21:17:02.106678 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:02.126902 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:02.129909 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:02.130077 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:02.136356 8115 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:45001
I20260928 21:17:02.136607 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:45001 every 8 connection(s)
I20260928 21:17:02.137269 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:02.138584 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8115
I20260928 21:17:02.138682 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:02.138927 5195 external_mini_cluster.cc:1428] Setting key f1d50f883440d18422aa0ee16f1bf3fa
I20260928 21:17:02.140175 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:02.146282 8183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4: Bootstrap starting.
Sep 28 21:17:02 dist-test-slave-rc9f krb5kdc[8102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:02.148540 8183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:02.149197 8183 log.cc:824] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:02.151230 8183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4: No bootstrap required, opened a new log
I20260928 21:17:02.152139 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:02.140150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.5.18.254:45001)
0928 21:17:02.140553 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:02.140570 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:02.140604 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:02.141226 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:02.141243 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:02.141552 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:02.141860 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:02.142711 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:02.143684 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:02.144784 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:02.145225 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:02.145327 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:02.147958 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:02.147962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:02.147997 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:02.148024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:02.149641 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:02.150042 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:02.150047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:02.150051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:02.150153 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:02.150404 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:02.150405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:02.150406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:02.150746 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:02.150876 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:02.151157 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 21:17:02.151411 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260928 21:17:02.154614 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:17:02.154790 8183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } }
I20260928 21:17:02.155035 8183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:02.155087 8183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d68c2ef70d94b59985f20cb315ce5b4, State: Initialized, Role: FOLLOWER
I20260928 21:17:02.155539 8183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } }
I20260928 21:17:02.155689 8183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:02.155747 8183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:02.155835 8183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:02.156826 8183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } }
I20260928 21:17:02.157128 8183 leader_election.cc:302] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d68c2ef70d94b59985f20cb315ce5b4; no voters:
I20260928 21:17:02.157397 8183 leader_election.cc:288] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:02.157480 8189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:02.157711 8189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d68c2ef70d94b59985f20cb315ce5b4, State: Running, Role: LEADER
I20260928 21:17:02.158048 8189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } }
I20260928 21:17:02.158614 8183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:02.159161 8189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d68c2ef70d94b59985f20cb315ce5b4. Latest consensus state: current_term: 1 leader_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } } }
I20260928 21:17:02.159274 8189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:02.159641 8190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d68c2ef70d94b59985f20cb315ce5b4" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45001 } } }
I20260928 21:17:02.159785 8190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:02.159777 8195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:02.162981 8195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:02.168223 8195 catalog_manager.cc:1347] Generated new cluster ID: 877729be0a7b47dd9c3f95d97df87d59
I20260928 21:17:02.168277 8195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:02.171097 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:02.166102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.5.18.254:45001)
0928 21:17:02.166204 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:02.166208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:02.166461 (+ 253us) server_negotiation.cc:407] Connection header received
0928 21:17:02.166593 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:02.166596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:02.166645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:02.166736 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:02.167229 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:02.167741 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:02.168807 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:02.169132 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:02.169198 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:02.170188 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:02.170192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:02.170195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:02.170637 (+ 442us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:02.170665 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:02.170683 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:02.170736 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:02.170904 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:17:02.170961 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:02.176836 8195 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:02.178010 8195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:02.190973 8195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d68c2ef70d94b59985f20cb315ce5b4: Generated new TSK 0
I20260928 21:17:02.191783 8195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:02.201483 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8115
2026-09-28T21:17:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:02 dist-test-slave-rc9f krb5kdc[8216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:02 dist-test-slave-rc9f krb5kdc[8216](info): set up 2 sockets
Sep 28 21:17:02 dist-test-slave-rc9f krb5kdc[8216](info): commencing operation
krb5kdc: starting...
W20260928 21:17:03.145450 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:03.254361 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43621
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:03.359952 8229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:03.360216 8229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:03.360272 8229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:03.364286 8229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:03.364341 8229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:03.364367 8229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:03.364389 8229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:03.364418 8229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:03.369503 8229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:43621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:03.370502 8229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:03.371438 8229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:03.379324 8235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:03.379271 8234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:03.379364 8229 server_base.cc:1034] running on GCE node
W20260928 21:17:03.379285 8237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:03.380041 8229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:03.381073 8229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:03.382239 8229 hybrid_clock.cc:648] HybridClock initialized: now 1790630223382215 us; error 34 us; skew 500 ppm
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:03.384707 8229 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:03.385512 8229 webserver.cc:460] Webserver started at http://127.5.18.254:39641/ using document root <none> and password file <none>
I20260928 21:17:03.386036 8229 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:03.386099 8229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:03.386279 8229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:03.388200 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "ac8e93dce7524baea50f64068229c619"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "e99c8f39ec6a2f00f3fa19037dcb9deb"
server_key_iv: "1457f96d33ea376be7984fa6741a3013"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.388747 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "ac8e93dce7524baea50f64068229c619"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "e99c8f39ec6a2f00f3fa19037dcb9deb"
server_key_iv: "1457f96d33ea376be7984fa6741a3013"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.392210 8229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:03.394449 8244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.395593 8229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:03.395711 8229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "ac8e93dce7524baea50f64068229c619"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "e99c8f39ec6a2f00f3fa19037dcb9deb"
server_key_iv: "1457f96d33ea376be7984fa6741a3013"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.395799 8229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:03.410914 8229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:03.413439 8229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:03.413645 8229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:03.419756 8229 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:43621
I20260928 21:17:03.420010 8296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:43621 every 8 connection(s)
I20260928 21:17:03.420630 8229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:03.423183 8297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:03.428228 8297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619: Bootstrap starting.
I20260928 21:17:03.429154 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8229
I20260928 21:17:03.429267 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:03.429544 5195 external_mini_cluster.cc:1428] Setting key c3b6a513c640052ad9d0332957e1b7c1
I20260928 21:17:03.430935 8297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:03.431613 8297 log.cc:824] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:03.433612 8297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619: No bootstrap required, opened a new log
I20260928 21:17:03.437536 8297 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } }
I20260928 21:17:03.437783 8297 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:03.437824 8297 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8e93dce7524baea50f64068229c619, State: Initialized, Role: FOLLOWER
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:03.438261 8297 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } }
I20260928 21:17:03.438382 8297 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:03.438443 8297 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:03.438532 8297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:03.439327 8297 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } }
I20260928 21:17:03.439636 8297 leader_election.cc:302] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac8e93dce7524baea50f64068229c619; no voters:
I20260928 21:17:03.439899 8297 leader_election.cc:288] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:03.439977 8302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:03.440138 8302 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [term 1 LEADER]: Becoming Leader. State: Replica: ac8e93dce7524baea50f64068229c619, State: Running, Role: LEADER
I20260928 21:17:03.440459 8302 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } }
I20260928 21:17:03.441092 8297 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:03.441988 8303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac8e93dce7524baea50f64068229c619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } } }
I20260928 21:17:03.442114 8303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:03.442412 8304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac8e93dce7524baea50f64068229c619. Latest consensus state: current_term: 1 leader_uuid: "ac8e93dce7524baea50f64068229c619" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e93dce7524baea50f64068229c619" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43621 } } }
I20260928 21:17:03.442523 8304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:03.442816 8311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:03.443658 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:03.430583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.5.18.254:43621)
0928 21:17:03.430938 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:03.430953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:03.430989 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:17:03.431604 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:03.431630 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:03.432000 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:03.432346 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:03.433139 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.434148 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.435282 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.435736 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.435839 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:03.438472 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:03.438478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:03.438496 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:03.438525 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:03.440219 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.440693 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.440698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.440704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.440816 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.441062 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.441065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.441067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.441512 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:03.441745 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:03.442431 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 21:17:03.442713 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260928 21:17:03.445596 8311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:03.450296 8311 catalog_manager.cc:1347] Generated new cluster ID: fd5112867b6d4c8fa9bd4d5647241628
I20260928 21:17:03.450354 8311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:03.458963 8311 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:03.460081 8311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:03.468963 8311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac8e93dce7524baea50f64068229c619: Generated new TSK 0
I20260928 21:17:03.469738 8311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:03.498265 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:03.604143 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:03.604384 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:03.604439 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:03.608356 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:03.608407 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:03.608484 8323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:03.613503 8323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:03.614486 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:03.615378 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:03.624202 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:03.624230 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:03.624245 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:03.624402 8323 server_base.cc:1034] running on GCE node
I20260928 21:17:03.624771 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:03.625320 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:03.626461 8323 hybrid_clock.cc:648] HybridClock initialized: now 1790630223626433 us; error 47 us; skew 500 ppm
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:03.628948 8323 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:03.629767 8323 webserver.cc:460] Webserver started at http://127.5.18.193:33149/ using document root <none> and password file <none>
I20260928 21:17:03.630249 8323 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:03.630303 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:03.630458 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:03.632340 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "f003f98d5f6c4611ad272d8e555a8202"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "8f235db7a2b697a572ad34c095562f4f"
server_key_iv: "743065b8480f43830873c2e4c4d4ab0e"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.632822 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "f003f98d5f6c4611ad272d8e555a8202"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "8f235db7a2b697a572ad34c095562f4f"
server_key_iv: "743065b8480f43830873c2e4c4d4ab0e"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.636346 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 21:17:03.638665 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.639946 8323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:03.640040 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "f003f98d5f6c4611ad272d8e555a8202"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "8f235db7a2b697a572ad34c095562f4f"
server_key_iv: "743065b8480f43830873c2e4c4d4ab0e"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.640142 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:03.660458 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:03.663434 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:03.663609 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:03.664191 8323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:03.664992 8323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:03.665036 8323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.665078 8323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:03.665105 8323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.673197 8323 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:38413
I20260928 21:17:03.673218 8451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:38413 every 8 connection(s)
I20260928 21:17:03.674093 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:03.684257 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8323
I20260928 21:17:03.684374 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:03.684675 5195 external_mini_cluster.cc:1428] Setting key a509779d889cbd8f58871eeabf7c0565
I20260928 21:17:03.687067 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:03.675793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:55337 (local address 127.5.18.254:43621)
0928 21:17:03.675917 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:03.675920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:03.676811 (+ 891us) server_negotiation.cc:407] Connection header received
0928 21:17:03.678315 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:03.678318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:03.678365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:03.678436 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:03.679526 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.680037 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.681137 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.681367 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.681443 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:03.683616 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:03.683619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:03.683621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:03.683645 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:03.685099 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.685632 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.685635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.685636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.685712 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.686064 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.686066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.686068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.686271 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:03.686365 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:03.686807 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 21:17:03.686918 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:17:03.687904 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:03.676167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43621 (local address 127.5.18.193:55337)
0928 21:17:03.676641 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:03.676695 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:03.678114 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:03.678588 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:03.678594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:03.678930 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:03.679404 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:03.679418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.680140 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.680143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:03.681021 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:03.681029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.681482 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.681485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:03.681666 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:03.682149 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:03.682164 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:03.683470 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:03.685201 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:03.685207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:03.685222 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:03.685497 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:03.685792 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:03.685795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:03.685797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:03.685974 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:03.686365 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:03.686369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:03.686584 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 21:17:03.686736 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 21:17:03.686967 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260928 21:17:03.689052 8452 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43621
I20260928 21:17:03.689270 8452 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:03.689754 8452 heartbeater.cc:511] Master 127.5.18.254:43621 requested a full tablet report, sending...
I20260928 21:17:03.691167 8261 ts_manager.cc:194] Registered new tserver with Master: f003f98d5f6c4611ad272d8e555a8202 (127.5.18.193:38413)
I20260928 21:17:03.692250 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:55337
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:03.708132 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:03.812808 8457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:03.813074 8457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:03.813129 8457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:03.817059 8457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:03.817116 8457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:03.817193 8457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:03.822461 8457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:03.823437 8457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:03.824404 8457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:03.833307 8463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:03.833469 8457 server_base.cc:1034] running on GCE node
W20260928 21:17:03.833325 8465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:03.833316 8462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:03.834005 8457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:03.834561 8457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:03.835713 8457 hybrid_clock.cc:648] HybridClock initialized: now 1790630223835697 us; error 30 us; skew 500 ppm
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:03.838158 8457 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:03.838949 8457 webserver.cc:460] Webserver started at http://127.5.18.194:38485/ using document root <none> and password file <none>
I20260928 21:17:03.839448 8457 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:03.839510 8457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:03.839694 8457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:03.841668 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "36b46978b40d41a1a09cbe5dbbfca371"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "3dc8741e64ecf5cdc6ac193e94e2550c"
server_key_iv: "a5bd1bc9843c7e6d154bbb4f181bd372"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.842221 8457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "36b46978b40d41a1a09cbe5dbbfca371"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "3dc8741e64ecf5cdc6ac193e94e2550c"
server_key_iv: "a5bd1bc9843c7e6d154bbb4f181bd372"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.845803 8457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:03.848029 8472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.849169 8457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:03.849278 8457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "36b46978b40d41a1a09cbe5dbbfca371"
format_stamp: "Formatted at 2026-09-28 21:17:03 on dist-test-slave-rc9f"
server_key: "3dc8741e64ecf5cdc6ac193e94e2550c"
server_key_iv: "a5bd1bc9843c7e6d154bbb4f181bd372"
server_key_version: "encryptionkey@0"
I20260928 21:17:03.849366 8457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:03.861671 8457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:03.864540 8457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:03.864720 8457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:03.865278 8457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:03.866081 8457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:03.866127 8457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.866179 8457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:03.866209 8457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:03.874322 8457 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:45249
I20260928 21:17:03.874354 8585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:45249 every 8 connection(s)
I20260928 21:17:03.875166 8457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:03.883304 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8457
I20260928 21:17:03.883397 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:03.883693 5195 external_mini_cluster.cc:1428] Setting key 17e25e344ec6dfe7ec863314bec87f26
Sep 28 21:17:03 dist-test-slave-rc9f krb5kdc[8216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:03.887871 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:03.876830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:54515 (local address 127.5.18.254:43621)
0928 21:17:03.876963 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:03.876966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:03.877773 (+ 807us) server_negotiation.cc:407] Connection header received
0928 21:17:03.879049 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:03.879052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:03.879108 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:03.879185 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:03.880314 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.880829 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.881960 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.882174 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.882261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:03.884639 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:03.884643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:03.884645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:03.884668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:03.885976 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.886431 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.886434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.886436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.886508 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:03.886807 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:03.886810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:03.886812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:03.886976 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:03.887068 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:03.887606 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 21:17:03.887721 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:17:03.888772 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:03.877195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43621 (local address 127.5.18.194:54515)
0928 21:17:03.877638 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:03.877681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:03.878853 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:03.879362 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:03.879369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:03.879711 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:03.880178 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:03.880191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.880940 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.880943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:03.881852 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:03.881862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:03.882304 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:03.882308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:03.882463 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:03.882951 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:03.882967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:03.884482 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:03.886077 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:03.886082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:03.886098 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:03.886342 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:03.886588 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:03.886591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:03.886593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:03.886725 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:03.887069 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:03.887076 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:03.887339 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 21:17:03.887543 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:17:03.887900 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20260928 21:17:03.889940 8586 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43621
I20260928 21:17:03.890163 8586 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:03.890626 8586 heartbeater.cc:511] Master 127.5.18.254:43621 requested a full tablet report, sending...
I20260928 21:17:03.891613 8261 ts_manager.cc:194] Registered new tserver with Master: 36b46978b40d41a1a09cbe5dbbfca371 (127.5.18.194:45249)
I20260928 21:17:03.892159 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:54515
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:03.908648 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:04.019690 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:04.019917 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:04.019956 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:04.023749 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:04.023795 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:04.023847 8591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:04.028703 8591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:04.029637 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:04.030678 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:04.039638 8596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:04.039739 8591 server_base.cc:1034] running on GCE node
W20260928 21:17:04.039753 8599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:04.039633 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:04.040345 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:04.040984 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:04.042143 8591 hybrid_clock.cc:648] HybridClock initialized: now 1790630224042123 us; error 30 us; skew 500 ppm
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:04.044750 8591 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:04.045646 8591 webserver.cc:460] Webserver started at http://127.5.18.195:45631/ using document root <none> and password file <none>
I20260928 21:17:04.046156 8591 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:04.046218 8591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:04.046392 8591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:04.048331 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "910ba7f2b6d2459caf2633af24b12aba"
format_stamp: "Formatted at 2026-09-28 21:17:04 on dist-test-slave-rc9f"
server_key: "1a3e7cc2df8357c7bfb77eb94a8f2dd8"
server_key_iv: "46193f1b7c4d67e88f344572401c41c2"
server_key_version: "encryptionkey@0"
I20260928 21:17:04.048847 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "910ba7f2b6d2459caf2633af24b12aba"
format_stamp: "Formatted at 2026-09-28 21:17:04 on dist-test-slave-rc9f"
server_key: "1a3e7cc2df8357c7bfb77eb94a8f2dd8"
server_key_iv: "46193f1b7c4d67e88f344572401c41c2"
server_key_version: "encryptionkey@0"
I20260928 21:17:04.052281 8591 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:04.054601 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:04.055831 8591 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:04.055941 8591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "910ba7f2b6d2459caf2633af24b12aba"
format_stamp: "Formatted at 2026-09-28 21:17:04 on dist-test-slave-rc9f"
server_key: "1a3e7cc2df8357c7bfb77eb94a8f2dd8"
server_key_iv: "46193f1b7c4d67e88f344572401c41c2"
server_key_version: "encryptionkey@0"
I20260928 21:17:04.056061 8591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:04.079190 8591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:04.081815 8591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:04.081987 8591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:04.082540 8591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:04.083308 8591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:04.083355 8591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:04.083406 8591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:04.083436 8591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:04.091379 8591 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:41177
I20260928 21:17:04.091413 8719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:41177 every 8 connection(s)
I20260928 21:17:04.092209 8591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:04.093834 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8591
I20260928 21:17:04.093937 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:04.094197 5195 external_mini_cluster.cc:1428] Setting key 301456e8f5a97ded959d549360a507f2
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:04.104550 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:04.093874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:41427 (local address 127.5.18.254:43621)
0928 21:17:04.094545 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:04.094549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:04.094845 (+ 296us) server_negotiation.cc:407] Connection header received
0928 21:17:04.096221 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:04.096224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:04.096276 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:04.096348 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:04.097443 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.098007 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.099123 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.099341 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.099425 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:04.101698 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:04.101702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:04.101703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:04.101728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:04.102957 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:04.103373 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:04.103375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:04.103376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:04.103442 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:04.103679 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:04.103681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:04.103682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:04.103829 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:04.103897 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:04.104346 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 21:17:04.104436 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576}
I20260928 21:17:04.105279 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:04.094272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43621 (local address 127.5.18.195:41427)
0928 21:17:04.094715 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:04.094756 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:04.096014 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:04.096517 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:04.096524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:04.096857 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:04.097314 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:04.097327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.098127 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.098130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:04.099019 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:04.099030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.099466 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.099469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:04.099627 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:04.100155 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:04.100175 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:04.101537 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:04.103051 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:04.103055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:04.103066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:04.103300 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:04.103502 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:04.103504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:04.103506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:04.103620 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:04.103935 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:04.103940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:04.104157 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 21:17:04.104308 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:17:04.104556 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260928 21:17:04.106215 8720 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43621
I20260928 21:17:04.106426 8720 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:04.106933 8720 heartbeater.cc:511] Master 127.5.18.254:43621 requested a full tablet report, sending...
I20260928 21:17:04.107892 8261 ts_manager.cc:194] Registered new tserver with Master: 910ba7f2b6d2459caf2633af24b12aba (127.5.18.195:41177)
I20260928 21:17:04.108407 8261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:41427
I20260928 21:17:04.119043 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630224, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630224, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:04.142095 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:04.134660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.5.18.254:43621)
0928 21:17:04.134799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:04.134805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:04.134922 (+ 117us) server_negotiation.cc:407] Connection header received
0928 21:17:04.135033 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:04.135036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:04.135080 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:04.135146 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:04.135764 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.136272 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.137013 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:04.137211 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:04.137272 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:04.139408 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:04.139412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:04.139414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:04.139438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:04.140627 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:04.141006 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:04.141008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:04.141010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:04.141081 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:04.141273 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:04.141275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:04.141276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:04.141481 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:04.141611 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:04.141835 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:17:04.141941 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 21:17:04.142462 8261 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56724
I20260928 21:17:04.143180 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8323
I20260928 21:17:04.148015 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8457
I20260928 21:17:04.152117 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8591
I20260928 21:17:04.156450 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8229
2026-09-28T21:17:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1956 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8732](info): set up 2 sockets
Sep 28 21:17:04 dist-test-slave-rc9f krb5kdc[8732](info): commencing operation
krb5kdc: starting...
W20260928 21:17:05.105312 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:05.214282 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38867
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:05.319078 8745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:05.319355 8745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:05.319408 8745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:05.323282 8745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:05.323333 8745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:05.323356 8745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:05.323378 8745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:05.323405 8745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:05.328747 8745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:05.329743 8745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:05.330693 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:05.338562 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.338639 8745 server_base.cc:1034] running on GCE node
W20260928 21:17:05.338515 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.338517 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.339311 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:05.340370 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:05.341522 8745 hybrid_clock.cc:648] HybridClock initialized: now 1790630225341504 us; error 31 us; skew 500 ppm
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:05.344053 8745 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:05.344869 8745 webserver.cc:460] Webserver started at http://127.5.18.254:43695/ using document root <none> and password file <none>
I20260928 21:17:05.345376 8745 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:05.345438 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:05.345685 8745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:05.347645 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "39230ed2cbcc400cac01e6a92efa138d"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "b0aeb1edab407a40783963befebd44cb"
server_key_iv: "6dc7e68afae35d47d7d35b7ca3532e63"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.348169 8745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "39230ed2cbcc400cac01e6a92efa138d"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "b0aeb1edab407a40783963befebd44cb"
server_key_iv: "6dc7e68afae35d47d7d35b7ca3532e63"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.351626 8745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:05.353819 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.354941 8745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:17:05.355053 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "39230ed2cbcc400cac01e6a92efa138d"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "b0aeb1edab407a40783963befebd44cb"
server_key_iv: "6dc7e68afae35d47d7d35b7ca3532e63"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.355137 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:05.369792 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:05.372355 8745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:05.372511 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:05.378669 8745 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38867
I20260928 21:17:05.378706 8812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38867 every 8 connection(s)
I20260928 21:17:05.379541 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:05.382047 8813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:05.386941 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d: Bootstrap starting.
I20260928 21:17:05.389103 8813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:05.389722 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8745
I20260928 21:17:05.389756 8813 log.cc:824] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:05.389827 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:05.390050 5195 external_mini_cluster.cc:1428] Setting key 9a849bc7816a506a52134994d4976ee1
I20260928 21:17:05.391561 8813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d: No bootstrap required, opened a new log
I20260928 21:17:05.395246 8813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } }
I20260928 21:17:05.395507 8813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:05.395574 8813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39230ed2cbcc400cac01e6a92efa138d, State: Initialized, Role: FOLLOWER
I20260928 21:17:05.395983 8813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } }
I20260928 21:17:05.396111 8813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:05.396201 8813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:05.396301 8813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:05.397285 8813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } }
I20260928 21:17:05.397668 8813 leader_election.cc:302] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39230ed2cbcc400cac01e6a92efa138d; no voters:
I20260928 21:17:05.397938 8813 leader_election.cc:288] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:05.398037 8818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:05.398267 8818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [term 1 LEADER]: Becoming Leader. State: Replica: 39230ed2cbcc400cac01e6a92efa138d, State: Running, Role: LEADER
I20260928 21:17:05.398550 8818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } }
I20260928 21:17:05.399003 8813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:05.400480 8819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39230ed2cbcc400cac01e6a92efa138d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } } }
I20260928 21:17:05.400599 8819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:05.400888 8820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39230ed2cbcc400cac01e6a92efa138d. Latest consensus state: current_term: 1 leader_uuid: "39230ed2cbcc400cac01e6a92efa138d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39230ed2cbcc400cac01e6a92efa138d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38867 } } }
I20260928 21:17:05.401649 8820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:05.401391 8832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:05.403638 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:05.390986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.5.18.254:38867)
0928 21:17:05.391341 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:05.391356 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:05.391387 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:17:05.392013 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:05.392032 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:05.392335 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:05.392589 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:05.393411 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.394366 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.395493 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.396011 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.396150 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:05.398864 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:05.398870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:05.398887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:05.398915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:05.400812 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.401333 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.401339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.401344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.401477 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.401826 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.401830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.401832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.402258 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:05.402440 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:05.402714 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 21:17:05.402968 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260928 21:17:05.404157 8832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:05.409041 8832 catalog_manager.cc:1347] Generated new cluster ID: a7d30ffd87b44a619e3f515b17356d01
I20260928 21:17:05.409087 8832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:05.423365 8832 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:05.424233 8832 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:05.427892 8832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39230ed2cbcc400cac01e6a92efa138d: Generated new TSK 0
I20260928 21:17:05.428642 8832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:05.454077 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:05.559229 8839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:05.559523 8839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:05.559587 8839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:05.564219 8839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:05.564288 8839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:05.564371 8839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:05.569703 8839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:05.570775 8839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:05.571820 8839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:05.581317 8845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.581326 8844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.581331 8847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.581601 8839 server_base.cc:1034] running on GCE node
I20260928 21:17:05.582031 8839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:05.582653 8839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:05.583809 8839 hybrid_clock.cc:648] HybridClock initialized: now 1790630225583790 us; error 34 us; skew 500 ppm
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:05.586436 8839 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:05.587249 8839 webserver.cc:460] Webserver started at http://127.5.18.193:36267/ using document root <none> and password file <none>
I20260928 21:17:05.587826 8839 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:05.587886 8839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:05.588047 8839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:05.590090 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "eb126e7b09474ded9023f282f7970731"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "2ce43f804f57e6dcb4af13cab1cbdc44"
server_key_iv: "22b91029b1f33294ccf00b486550faaa"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.590574 8839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "eb126e7b09474ded9023f282f7970731"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "2ce43f804f57e6dcb4af13cab1cbdc44"
server_key_iv: "22b91029b1f33294ccf00b486550faaa"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.594364 8839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:17:05.596755 8854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.597992 8839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:05.598088 8839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "eb126e7b09474ded9023f282f7970731"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "2ce43f804f57e6dcb4af13cab1cbdc44"
server_key_iv: "22b91029b1f33294ccf00b486550faaa"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.598176 8839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:05.607941 8839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:05.610709 8839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:05.610852 8839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:05.611356 8839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:05.612123 8839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:05.612174 8839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.612215 8839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:05.612234 8839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.620141 8839 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:44805
I20260928 21:17:05.620162 8967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:44805 every 8 connection(s)
I20260928 21:17:05.620978 8839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:05.629769 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8839
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:05.629886 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:05.630167 5195 external_mini_cluster.cc:1428] Setting key 06ce15aa657dccf69e8539e09be1f66e
I20260928 21:17:05.633886 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:05.622666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56865 (local address 127.5.18.254:38867)
0928 21:17:05.622818 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:05.622823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:05.623618 (+ 795us) server_negotiation.cc:407] Connection header received
0928 21:17:05.625055 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:05.625057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:05.625104 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:05.625173 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:05.626288 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.626823 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.627885 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.628113 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.628187 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:05.630453 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:05.630456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:05.630459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:05.630488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:05.631917 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.632380 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.632384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.632386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.632483 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.632812 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.632815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.632817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.632989 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:05.633074 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:05.633543 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 21:17:05.633728 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:17:05.634646 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:05.623059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38867 (local address 127.5.18.193:56865)
0928 21:17:05.623475 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:05.623514 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:05.624853 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:05.625320 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:05.625327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:05.625681 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:05.626168 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:05.626181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.626907 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.626909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:05.627793 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:05.627801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.628223 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.628225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:05.628358 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:05.628886 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:05.628900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:05.630309 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:05.632018 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:05.632023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:05.632037 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:05.632289 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:05.632578 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:05.632581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:05.632583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:05.632725 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:05.633083 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:05.633089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:05.633324 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:17:05.633487 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 21:17:05.633827 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260928 21:17:05.635748 8968 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38867
I20260928 21:17:05.635978 8968 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:05.636428 8968 heartbeater.cc:511] Master 127.5.18.254:38867 requested a full tablet report, sending...
I20260928 21:17:05.637835 8777 ts_manager.cc:194] Registered new tserver with Master: eb126e7b09474ded9023f282f7970731 (127.5.18.193:44805)
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:17:05.638947 8777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:56865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:05.654422 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:05.760212 8973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:05.760488 8973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:05.760530 8973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:05.764539 8973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:05.764606 8973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:05.764667 8973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:05.769987 8973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:05.770974 8973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:05.771953 8973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:05.781087 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.781060 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.781070 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.781118 8973 server_base.cc:1034] running on GCE node
I20260928 21:17:05.781749 8973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:05.782337 8973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:05.783499 8973 hybrid_clock.cc:648] HybridClock initialized: now 1790630225783487 us; error 28 us; skew 500 ppm
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:05.786032 8973 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:05.786824 8973 webserver.cc:460] Webserver started at http://127.5.18.194:37757/ using document root <none> and password file <none>
I20260928 21:17:05.787338 8973 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:05.787400 8973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:05.787596 8973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:05.789783 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "04ce062a47d64dd1abe9381ad5672da2"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "c783ee34539058830f7367174229f825"
server_key_iv: "b44e77c0ec1233a4440d4d8af086cd7b"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.790344 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "04ce062a47d64dd1abe9381ad5672da2"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "c783ee34539058830f7367174229f825"
server_key_iv: "b44e77c0ec1233a4440d4d8af086cd7b"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.794068 8973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:05.796417 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.797648 8973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:05.797749 8973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "04ce062a47d64dd1abe9381ad5672da2"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "c783ee34539058830f7367174229f825"
server_key_iv: "b44e77c0ec1233a4440d4d8af086cd7b"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.797837 8973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:05.813444 8973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:05.815979 8973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:05.816138 8973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:05.816656 8973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:05.817396 8973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:05.817440 8973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.817488 8973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:05.817518 8973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.825317 8973 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:39039
I20260928 21:17:05.825341 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:39039 every 8 connection(s)
I20260928 21:17:05.826162 8973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:05.829407 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 8973
I20260928 21:17:05.829504 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:05.829779 5195 external_mini_cluster.cc:1428] Setting key eda9c41e79ba72a925594d3d6803d20f
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:05.838780 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:05.827740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45037 (local address 127.5.18.254:38867)
0928 21:17:05.827872 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:05.827875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:05.828625 (+ 750us) server_negotiation.cc:407] Connection header received
0928 21:17:05.830046 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:05.830050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:05.830105 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:05.830172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:05.831270 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.831786 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.832855 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.833071 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.833155 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:05.835591 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:05.835595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:05.835597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:05.835619 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:05.836909 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.837351 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.837355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.837357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.837428 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:05.837787 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:05.837791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:05.837793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:05.838015 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:05.838104 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:05.838551 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 21:17:05.838653 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:17:05.839479 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:05.828086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38867 (local address 127.5.18.194:45037)
0928 21:17:05.828492 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:05.828530 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:05.829840 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:05.830348 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:05.830355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:05.830681 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:05.831144 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:05.831157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.831893 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.831897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:05.832748 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:05.832756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:05.833194 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:05.833198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:05.833348 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:05.833913 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:05.833928 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:05.835432 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:05.837011 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:05.837016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:05.837031 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:05.837262 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:05.837508 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:05.837511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:05.837513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:05.837684 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:05.838130 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:05.838135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:05.838350 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 21:17:05.838503 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:17:05.838733 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:05.840524 9102 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38867
I20260928 21:17:05.840768 9102 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:05.841307 9102 heartbeater.cc:511] Master 127.5.18.254:38867 requested a full tablet report, sending...
I20260928 21:17:05.842353 8777 ts_manager.cc:194] Registered new tserver with Master: 04ce062a47d64dd1abe9381ad5672da2 (127.5.18.194:39039)
I20260928 21:17:05.842880 8777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:45037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:05.854111 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:05.958741 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:05.958992 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:05.959051 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:05.963068 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:05.963130 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:05.963212 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:05.968333 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:05.969347 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:05.970363 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:05.979532 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:05.979528 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.979710 9107 server_base.cc:1034] running on GCE node
W20260928 21:17:05.979544 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:05.980244 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:05.980849 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:05.981992 9107 hybrid_clock.cc:648] HybridClock initialized: now 1790630225981975 us; error 35 us; skew 500 ppm
Sep 28 21:17:05 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:05.985105 9107 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:05.986107 9107 webserver.cc:460] Webserver started at http://127.5.18.195:35781/ using document root <none> and password file <none>
I20260928 21:17:05.986720 9107 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:05.986783 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:05.986984 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:05.989491 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "45b078c6eb244d7bb2033868fc85afed"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "581f54ad6bb0a81f65b8cf77086e47ec"
server_key_iv: "15f7669502dd15f42c1874bebe86e0f3"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.990113 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "45b078c6eb244d7bb2033868fc85afed"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "581f54ad6bb0a81f65b8cf77086e47ec"
server_key_iv: "15f7669502dd15f42c1874bebe86e0f3"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.994619 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:17:05.997184 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:05.998365 9107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:05.998471 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "45b078c6eb244d7bb2033868fc85afed"
format_stamp: "Formatted at 2026-09-28 21:17:05 on dist-test-slave-rc9f"
server_key: "581f54ad6bb0a81f65b8cf77086e47ec"
server_key_iv: "15f7669502dd15f42c1874bebe86e0f3"
server_key_version: "encryptionkey@0"
I20260928 21:17:05.998559 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:06.012212 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:06.015059 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:06.015220 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:06.015743 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:06.016492 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:06.016535 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:06.016585 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:06.016609 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:06.024317 9107 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:34081
I20260928 21:17:06.024362 9235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:34081 every 8 connection(s)
I20260928 21:17:06.025152 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:06.029497 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9107
I20260928 21:17:06.029627 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:06.029862 5195 external_mini_cluster.cc:1428] Setting key 72357e87419a82354f92e55d22446dc6
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:06.037771 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:06.026860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46219 (local address 127.5.18.254:38867)
0928 21:17:06.026988 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:06.026992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:06.027795 (+ 803us) server_negotiation.cc:407] Connection header received
0928 21:17:06.029050 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:06.029053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:06.029101 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:06.029171 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:06.030472 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.031110 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.032207 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.032416 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.032500 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:06.034812 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:06.034815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:06.034817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:06.034840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:06.035899 (+ 1059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.036379 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.036383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.036385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.036458 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.036745 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.036748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.036750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.036917 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:06.037017 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:06.037461 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 21:17:06.037615 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:06.038552 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:06.027240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38867 (local address 127.5.18.195:46219)
0928 21:17:06.027655 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:06.027695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:06.028871 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:06.029334 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:06.029341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:06.029770 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:06.030342 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:06.030358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.031225 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.031228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:06.032098 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:06.032107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.032541 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.032545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:06.032716 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:06.033212 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:06.033227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:06.034659 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:06.036035 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:06.036040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:06.036055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:06.036291 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:06.036534 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:06.036537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:06.036539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:06.036664 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:06.037014 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:06.037020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:06.037249 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 21:17:06.037406 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 21:17:06.037775 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260928 21:17:06.039511 9236 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38867
I20260928 21:17:06.039743 9236 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:06.040191 9236 heartbeater.cc:511] Master 127.5.18.254:38867 requested a full tablet report, sending...
I20260928 21:17:06.041116 8777 ts_manager.cc:194] Registered new tserver with Master: 45b078c6eb244d7bb2033868fc85afed (127.5.18.195:34081)
I20260928 21:17:06.041682 8777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:46219
I20260928 21:17:06.043200 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[8732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:17:06.066642 9243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:06.058224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.5.18.193:44805)
0928 21:17:06.058522 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:06.058531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:06.058553 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:17:06.058599 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:06.058604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:06.058804 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:06.058922 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:06.059565 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.060333 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.061109 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.061469 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.061545 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:06.063646 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:06.063654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:06.063666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:06.063687 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:06.065184 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.065505 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.065509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.065511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.065624 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.065822 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.065824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.065825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.066060 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:06.066223 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:06.066336 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:17:06.066462 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":109,"threads_started":1}
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[8732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:06.076444 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:06.069318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.5.18.254:38867)
0928 21:17:06.069429 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:06.069432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:06.069500 (+ 68us) server_negotiation.cc:407] Connection header received
0928 21:17:06.069654 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:06.069656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:06.069697 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:06.069755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:06.070240 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.070722 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.071500 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:06.071714 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:06.071777 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:06.073862 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:06.073866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:06.073868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:06.073890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:06.075119 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.075484 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.075486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.075487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.075552 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:06.075721 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:06.075723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:06.075724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:06.075908 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:06.076005 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:06.076220 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:17:06.076303 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:17:06.076790 8777 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59360
I20260928 21:17:06.077324 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8839
I20260928 21:17:06.082209 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8973
I20260928 21:17:06.086974 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9107
I20260928 21:17:06.091104 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 8745
2026-09-28T21:17:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1934 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[9248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[9248](info): set up 2 sockets
Sep 28 21:17:06 dist-test-slave-rc9f krb5kdc[9248](info): commencing operation
krb5kdc: starting...
W20260928 21:17:07.037367 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:07.148411 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:36541
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:07.252990 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:07.253269 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:07.253311 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:07.257400 9261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:07.257490 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:07.257509 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:07.257526 9261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:07.257540 9261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:07.262970 9261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:07.264075 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:07.265098 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:07.273136 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:07.273218 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.273401 9261 server_base.cc:1034] running on GCE node
W20260928 21:17:07.273208 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.274001 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:07.275060 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:07.276207 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790630227276181 us; error 41 us; skew 500 ppm
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:07.285812 9261 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:07.286634 9261 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:07.286702 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:07.286906 9261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:07.288944 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "723796c7ef2048d380b3066948f403c2"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7d513dd1e6b3b1fb612663352076276b"
server_key_iv: "1a40c8dc46eabf23bf8c3178bbe6ca1d"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.289472 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "723796c7ef2048d380b3066948f403c2"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7d513dd1e6b3b1fb612663352076276b"
server_key_iv: "1a40c8dc46eabf23bf8c3178bbe6ca1d"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.293002 9261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 21:17:07.295323 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.296499 9261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:07.296610 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "723796c7ef2048d380b3066948f403c2"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7d513dd1e6b3b1fb612663352076276b"
server_key_iv: "1a40c8dc46eabf23bf8c3178bbe6ca1d"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.296700 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:07.309927 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:07.312466 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:07.312630 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:07.318979 9261 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:36541
I20260928 21:17:07.319196 9327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:36541 every 8 connection(s)
I20260928 21:17:07.319574 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:07.322078 9328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:07.324010 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9261
I20260928 21:17:07.324116 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:07.324365 5195 external_mini_cluster.cc:1428] Setting key 577b17fbcc999bd14b0c491f0a5c0d41
I20260928 21:17:07.328132 9328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: Bootstrap starting.
I20260928 21:17:07.330657 9328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:07.331391 9328 log.cc:824] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:07.333359 9328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: No bootstrap required, opened a new log
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:07.336844 9328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } }
I20260928 21:17:07.337059 9328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:07.337114 9328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723796c7ef2048d380b3066948f403c2, State: Initialized, Role: FOLLOWER
I20260928 21:17:07.337514 9328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } }
I20260928 21:17:07.337675 9328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:07.337736 9328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:07.337829 9328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:07.338114 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:07.325726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.5.18.254:36541)
0928 21:17:07.326120 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:07.326142 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:07.326185 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:17:07.326988 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:07.327012 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:07.327440 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:07.327793 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:07.328457 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.329910 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.330795 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.331298 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.331456 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:07.333935 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:07.333940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:07.333953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:07.333978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:07.335475 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.335854 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.335859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.335864 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.335967 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.336203 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.336206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.336208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.336573 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:07.336758 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:07.336996 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:17:07.337282 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260928 21:17:07.338697 9328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } }
I20260928 21:17:07.339016 9328 leader_election.cc:302] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723796c7ef2048d380b3066948f403c2; no voters:
I20260928 21:17:07.339320 9328 leader_election.cc:288] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:07.339394 9333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:07.339614 9333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [term 1 LEADER]: Becoming Leader. State: Replica: 723796c7ef2048d380b3066948f403c2, State: Running, Role: LEADER
I20260928 21:17:07.339897 9333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } }
I20260928 21:17:07.340554 9328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:07.340994 9335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723796c7ef2048d380b3066948f403c2. Latest consensus state: current_term: 1 leader_uuid: "723796c7ef2048d380b3066948f403c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } } }
I20260928 21:17:07.341109 9335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:07.341341 9334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "723796c7ef2048d380b3066948f403c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723796c7ef2048d380b3066948f403c2" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 36541 } } }
I20260928 21:17:07.341439 9334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:07.341463 9338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:07.344439 9338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:07.350507 9338 catalog_manager.cc:1347] Generated new cluster ID: bf368cd4a7a147b69c80b344cc85f286
I20260928 21:17:07.350571 9338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:07.364198 9338 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:07.366003 9338 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:07.374604 9338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: Generated new TSK 0
I20260928 21:17:07.375339 9338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:07.408643 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:07.547963 9354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:07.548238 9354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:07.548300 9354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:07.552392 9354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:07.552481 9354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:07.552565 9354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:07.557993 9354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:07.559065 9354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:07.560077 9354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:07.569945 9362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.570106 9354 server_base.cc:1034] running on GCE node
W20260928 21:17:07.569945 9360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:07.569938 9359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.570691 9354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:07.571314 9354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:07.572485 9354 hybrid_clock.cc:648] HybridClock initialized: now 1790630227572450 us; error 52 us; skew 500 ppm
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:07.575245 9354 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:07.575946 9354 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:07.576013 9354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:07.576208 9354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:07.578277 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "be1f330437544a65b02fbf3b95dcff44"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "ef3513a30b31c775fb999cba01b3a4cf"
server_key_iv: "b0aa53db288910923e7e470e66cf9a3b"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.578806 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "be1f330437544a65b02fbf3b95dcff44"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "ef3513a30b31c775fb999cba01b3a4cf"
server_key_iv: "b0aa53db288910923e7e470e66cf9a3b"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.582410 9354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:07.584721 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.586023 9354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:07.586135 9354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "be1f330437544a65b02fbf3b95dcff44"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "ef3513a30b31c775fb999cba01b3a4cf"
server_key_iv: "b0aa53db288910923e7e470e66cf9a3b"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.586227 9354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:07.601195 9354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:07.604182 9354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:07.604331 9354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:07.604856 9354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:07.605672 9354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:07.605719 9354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.605757 9354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:07.605777 9354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.613722 9354 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:43545
I20260928 21:17:07.613762 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:43545 every 8 connection(s)
I20260928 21:17:07.614288 9354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:07.616001 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9354
I20260928 21:17:07.616102 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:07.616372 5195 external_mini_cluster.cc:1428] Setting key c51f3989211bed5fd1b3b6902b998ee5
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:07.629037 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:07.616008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:36491 (local address 127.5.18.254:36541)
0928 21:17:07.616156 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:07.616160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:07.617082 (+ 922us) server_negotiation.cc:407] Connection header received
0928 21:17:07.618539 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:07.618542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:07.618593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:07.618663 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:07.620181 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.620829 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.622543 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.622784 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.622873 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:07.625614 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:07.625619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:07.625622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:07.625649 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:07.626942 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.627476 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.627480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.627481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.627554 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.627898 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.627901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.627903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.628094 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:07.628193 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:07.628736 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 21:17:07.628860 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:17:07.630045 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:07.616425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36541 (local address 127.5.18.193:36491)
0928 21:17:07.616916 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:07.616982 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:07.618343 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:07.618830 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:07.618837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:07.619318 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:07.620015 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:07.620035 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.620966 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.620969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:07.622415 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:07.622427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.622940 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.622953 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:07.623161 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:07.623783 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:07.623804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:07.625427 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:07.627047 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:07.627052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:07.627076 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:07.627379 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:07.627633 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:07.627636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:07.627638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:07.627813 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:07.628221 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:07.628226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:07.628495 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 21:17:07.628674 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 21:17:07.628958 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
I20260928 21:17:07.631366 9482 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36541
I20260928 21:17:07.631605 9482 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:07.632087 9482 heartbeater.cc:511] Master 127.5.18.254:36541 requested a full tablet report, sending...
I20260928 21:17:07.633740 9292 ts_manager.cc:194] Registered new tserver with Master: be1f330437544a65b02fbf3b95dcff44 (127.5.18.193:43545)
I20260928 21:17:07.635360 9292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:36491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:07.643852 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:07.751237 9487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:07.751473 9487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:07.751516 9487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:07.755445 9487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:07.755494 9487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:07.755556 9487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:07.760563 9487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:07.761493 9487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:07.762457 9487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:07.771166 9493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.771284 9487 server_base.cc:1034] running on GCE node
W20260928 21:17:07.771153 9492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:07.771153 9495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.771799 9487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:07.772389 9487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:07.773551 9487 hybrid_clock.cc:648] HybridClock initialized: now 1790630227773522 us; error 41 us; skew 500 ppm
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:07.776018 9487 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:07.776686 9487 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:07.776752 9487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:07.776926 9487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:07.778862 9487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "cc3f443713a040cd8d4ca684dc13be23"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "dce0ffaa1ada2c7855546240f1f70905"
server_key_iv: "e291cdc3afa0495d3ffdd31b97778276"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.779361 9487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "cc3f443713a040cd8d4ca684dc13be23"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "dce0ffaa1ada2c7855546240f1f70905"
server_key_iv: "e291cdc3afa0495d3ffdd31b97778276"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.782806 9487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:07.785029 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.786242 9487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:07.786353 9487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "cc3f443713a040cd8d4ca684dc13be23"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "dce0ffaa1ada2c7855546240f1f70905"
server_key_iv: "e291cdc3afa0495d3ffdd31b97778276"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.786451 9487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:07.806031 9487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:07.808722 9487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:07.808884 9487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:07.809465 9487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:07.810279 9487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:07.810324 9487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.810375 9487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:07.810403 9487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.818315 9487 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:39451
I20260928 21:17:07.818334 9614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:39451 every 8 connection(s)
I20260928 21:17:07.818850 9487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:07.819697 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9487
I20260928 21:17:07.819815 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:07.820106 5195 external_mini_cluster.cc:1428] Setting key f6cad58030f006527f7e486adbdd232f
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:07.831614 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:07.820663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43733 (local address 127.5.18.254:36541)
0928 21:17:07.820918 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:07.820922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:07.821724 (+ 802us) server_negotiation.cc:407] Connection header received
0928 21:17:07.822999 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:07.823002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:07.823054 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:07.823126 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:07.824276 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.824791 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.825950 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.826170 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.826257 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:07.828555 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:07.828558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:07.828560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:07.828590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:07.829860 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.830318 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.830320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.830321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.830385 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:07.830668 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:07.830671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:07.830672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:07.830853 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:07.830935 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:07.831382 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 21:17:07.831492 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260928 21:17:07.832458 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:07.821066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36541 (local address 127.5.18.194:43733)
0928 21:17:07.821553 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:07.821633 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:07.822807 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:07.823306 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:07.823314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:07.823664 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:07.824142 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:07.824156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.824908 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.824912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:07.825843 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:07.825852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:07.826305 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:07.826309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:07.826464 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:07.826980 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:07.826996 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:07.828405 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:07.829976 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:07.829981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:07.829993 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:07.830246 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:07.830448 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:07.830450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:07.830451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:07.830569 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:07.830968 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:07.830973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:07.831194 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 21:17:07.831345 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:17:07.831590 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":117,"threads_started":1}
I20260928 21:17:07.833590 9615 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36541
I20260928 21:17:07.833832 9615 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:07.834327 9615 heartbeater.cc:511] Master 127.5.18.254:36541 requested a full tablet report, sending...
I20260928 21:17:07.835438 9292 ts_manager.cc:194] Registered new tserver with Master: cc3f443713a040cd8d4ca684dc13be23 (127.5.18.194:39451)
I20260928 21:17:07.836007 9292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:43733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:07.848613 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:07.956951 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:07.957206 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:07.957268 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:07.961229 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:07.961292 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:07.961375 9620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:07.966707 9620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:07.967679 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:07.968654 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:07.977675 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:07.977712 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:07.977686 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:07.978214 9620 server_base.cc:1034] running on GCE node
I20260928 21:17:07.978570 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:07.979158 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:07.980325 9620 hybrid_clock.cc:648] HybridClock initialized: now 1790630227980300 us; error 39 us; skew 500 ppm
Sep 28 21:17:07 dist-test-slave-rc9f krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:07.983027 9620 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:07.983706 9620 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:07.983770 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:07.983955 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:07.985906 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "df07a9f02eb74044b975811ec6b9e82f"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7a7e64faf5a7d5843793f3186645dda2"
server_key_iv: "75936ef2608d16a6930ddc8eb397156e"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.986416 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "df07a9f02eb74044b975811ec6b9e82f"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7a7e64faf5a7d5843793f3186645dda2"
server_key_iv: "75936ef2608d16a6930ddc8eb397156e"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.990043 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:07.992416 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:07.993542 9620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:07.993672 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "df07a9f02eb74044b975811ec6b9e82f"
format_stamp: "Formatted at 2026-09-28 21:17:07 on dist-test-slave-rc9f"
server_key: "7a7e64faf5a7d5843793f3186645dda2"
server_key_iv: "75936ef2608d16a6930ddc8eb397156e"
server_key_version: "encryptionkey@0"
I20260928 21:17:07.993758 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:08.007257 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:08.010147 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:08.010310 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:08.010839 9620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:08.011603 9620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:08.011665 9620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.011718 9620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:08.011741 9620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.019479 9620 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:32977
I20260928 21:17:08.019502 9747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:32977 every 8 connection(s)
I20260928 21:17:08.020001 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:08.027603 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9620
I20260928 21:17:08.027725 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:08.028021 5195 external_mini_cluster.cc:1428] Setting key 50544ed0df8dffae1db9d9324c6ff788
Sep 28 21:17:08 dist-test-slave-rc9f krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630227, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:08.032219 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.021611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:48957 (local address 127.5.18.254:36541)
0928 21:17:08.021732 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.021735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.022511 (+ 776us) server_negotiation.cc:407] Connection header received
0928 21:17:08.023732 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.023735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.023779 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.023849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.024898 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.025397 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.026471 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.026678 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.026744 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.029280 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.029285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.029288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:08.029313 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.030640 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:08.031064 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:08.031067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:08.031068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:08.031132 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:08.031377 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:08.031379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:08.031380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:08.031543 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.031611 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.032011 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.032103 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:17:08.033001 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.021943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:36541 (local address 127.5.18.195:48957)
0928 21:17:08.022362 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.022401 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.023544 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.023994 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.024000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.024336 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.024789 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.024803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.025482 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.025485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.026383 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.026392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.026777 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.026780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.026913 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.027443 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:08.027463 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.029135 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.030735 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:08.030739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:08.030751 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:08.030991 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:08.031193 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:08.031195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:08.031197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:08.031316 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:08.031615 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.031620 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.031837 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.031981 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.032218 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
I20260928 21:17:08.034003 9748 heartbeater.cc:347] Connected to a master server at 127.5.18.254:36541
I20260928 21:17:08.034206 9748 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:08.034622 9748 heartbeater.cc:511] Master 127.5.18.254:36541 requested a full tablet report, sending...
I20260928 21:17:08.035526 9292 ts_manager.cc:194] Registered new tserver with Master: df07a9f02eb74044b975811ec6b9e82f (127.5.18.195:32977)
I20260928 21:17:08.036070 9292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:48957
I20260928 21:17:08.041482 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:17:08.049708 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.043241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.5.18.254:36541)
0928 21:17:08.043354 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.043358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.043510 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:17:08.043635 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.043637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.043676 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.043736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.044389 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.044881 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.045712 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.045925 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.046008 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.047178 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.047182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.047184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:08.047205 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.048224 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:08.048591 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:08.048595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:08.048597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:08.048667 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:08.048890 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:08.048893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:08.048895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:08.049102 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.049217 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.049426 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.049527 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:17:08.052013 9292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:08.054836 9292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:08.067830 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.062209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:39451 (local address 127.0.0.1:55540)
0928 21:17:08.062518 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.062569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.062714 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.063697 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.063703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.063725 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:08.063861 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.063867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.065255 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.065257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.066409 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.066418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.067305 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.067308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.067441 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.067481 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.067572 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.067687 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260928 21:17:08.067863 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.062303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.5.18.194:39451)
0928 21:17:08.063204 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.063221 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.063244 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:17:08.063321 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.063327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.063544 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.063711 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:08.063973 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.065123 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.066517 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.067193 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.067295 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.067434 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.067634 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.067732 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":99,"threads_started":1}
I20260928 21:17:08.068307 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.062739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:43545 (local address 127.0.0.1:37554)
0928 21:17:08.063521 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.063534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.063627 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.064114 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.064116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.064146 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:08.064281 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.064287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.065616 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.065619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.066757 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.066765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.067929 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.067933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.068029 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.068050 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.068112 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.068173 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":23,"spinlock_wait_cycles":13184,"thread_start_us":93,"threads_started":1}
I20260928 21:17:08.068787 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.062884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.5.18.193:43545)
0928 21:17:08.063285 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.063296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.063718 (+ 422us) server_negotiation.cc:407] Connection header received
0928 21:17:08.063797 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.063802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.063982 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.064137 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:08.064385 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.065431 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.066875 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.067647 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.067742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.067868 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.068552 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.068647 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":106,"threads_started":1}
I20260928 21:17:08.069109 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.062831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:32977 (local address 127.0.0.1:55868)
0928 21:17:08.063411 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.063423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.063524 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.064117 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.064119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.064144 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:08.064280 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.064287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.066561 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.066564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.067458 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.067466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.068582 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.068586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.068670 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.068692 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.068752 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.068813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":37,"spinlock_wait_cycles":6272,"thread_start_us":95,"threads_started":1}
I20260928 21:17:08.069752 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.063228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.5.18.195:32977)
0928 21:17:08.063587 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.063598 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.063617 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:17:08.063684 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.063689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.063859 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.063990 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:08.065286 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.066421 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.067573 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.068485 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.068577 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.069384 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.069432 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.069644 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":106,"threads_started":1}
I20260928 21:17:08.071231 9416 tablet_service.cc:1467] Processing CreateTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 (DEFAULT_TABLE table=test-table [id=c8cf827aa93247f1af996912af1fdebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.071231 9549 tablet_service.cc:1467] Processing CreateTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 (DEFAULT_TABLE table=test-table [id=c8cf827aa93247f1af996912af1fdebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.071867 9682 tablet_service.cc:1467] Processing CreateTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 (DEFAULT_TABLE table=test-table [id=c8cf827aa93247f1af996912af1fdebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.072342 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a73b47ff86d429a9a0d025d0d159085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.072342 9416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a73b47ff86d429a9a0d025d0d159085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.072800 9682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a73b47ff86d429a9a0d025d0d159085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.078327 9765 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Bootstrap starting.
I20260928 21:17:08.079638 9767 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Bootstrap starting.
I20260928 21:17:08.079929 9766 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: Bootstrap starting.
I20260928 21:17:08.080582 9765 tablet_bootstrap.cc:654] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.081600 9766 tablet_bootstrap.cc:654] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.081701 9767 tablet_bootstrap.cc:654] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.081848 9765 log.cc:824] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.082703 9766 log.cc:824] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.082795 9767 log.cc:824] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.083992 9765 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: No bootstrap required, opened a new log
I20260928 21:17:08.084198 9765 ts_tablet_manager.cc:1397] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:17:08.084406 9766 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: No bootstrap required, opened a new log
I20260928 21:17:08.084589 9766 ts_tablet_manager.cc:1397] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:17:08.084753 9767 tablet_bootstrap.cc:492] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: No bootstrap required, opened a new log
I20260928 21:17:08.084926 9767 ts_tablet_manager.cc:1397] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 21:17:08.087359 9766 raft_consensus.cc:348] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.087586 9766 raft_consensus.cc:374] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.087643 9766 raft_consensus.cc:729] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3f443713a040cd8d4ca684dc13be23, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.088037 9766 consensus_queue.cc:260] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.087988 9765 raft_consensus.cc:348] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.088258 9765 raft_consensus.cc:374] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.088310 9765 raft_consensus.cc:729] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df07a9f02eb74044b975811ec6b9e82f, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.088481 9767 raft_consensus.cc:348] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.088714 9767 raft_consensus.cc:374] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.088745 9615 heartbeater.cc:503] Master 127.5.18.254:36541 was elected leader, sending a full tablet report...
I20260928 21:17:08.088759 9767 raft_consensus.cc:729] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1f330437544a65b02fbf3b95dcff44, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.088713 9765 consensus_queue.cc:260] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.088992 9766 ts_tablet_manager.cc:1428] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260928 21:17:08.089180 9767 consensus_queue.cc:260] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.089679 9765 ts_tablet_manager.cc:1428] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260928 21:17:08.089987 9748 heartbeater.cc:503] Master 127.5.18.254:36541 was elected leader, sending a full tablet report...
I20260928 21:17:08.091377 9767 ts_tablet_manager.cc:1428] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 21:17:08.091683 9482 heartbeater.cc:503] Master 127.5.18.254:36541 was elected leader, sending a full tablet report...
W20260928 21:17:08.115265 9483 tablet.cc:2369] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:08.116106 9772 raft_consensus.cc:482] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:08.116266 9772 raft_consensus.cc:504] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.117300 9772 leader_election.cc:288] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc3f443713a040cd8d4ca684dc13be23 (127.5.18.194:39451), be1f330437544a65b02fbf3b95dcff44 (127.5.18.193:43545)
I20260928 21:17:08.121506 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.117640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:60069 (local address 127.5.18.193:43545)
0928 21:17:08.117748 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.117752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.117766 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:08.117876 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.117879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.117924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.117985 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:08.118257 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.119013 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.120507 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.121000 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.121102 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.121152 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.121316 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.121381 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:08.121558 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.117663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:43585 (local address 127.5.18.194:39451)
0928 21:17:08.117794 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.117798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.118020 (+ 222us) server_negotiation.cc:407] Connection header received
0928 21:17:08.118107 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.118109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.118150 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.118203 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:08.118419 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.119146 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.120537 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.121054 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.121176 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.121224 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.121382 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.121449 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:17:08.121732 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.117515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:39451 (local address 127.5.18.195:43585)
0928 21:17:08.117936 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.117946 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.118034 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.118232 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.118234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.118253 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:08.118337 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.118341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.119228 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.119231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.120381 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.120391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.121168 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.121172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.121273 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.121293 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.121355 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.121423 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":147,"thread_start_us":93,"threads_started":1}
I20260928 21:17:08.121783 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.117515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:43545 (local address 127.5.18.195:60069)
0928 21:17:08.117651 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.117666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.117755 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.118020 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.118023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.118035 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:08.118156 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.118161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.119116 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.119119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.120379 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:08.120391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.121102 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.121105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:08.121213 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.121230 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.121292 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.121362 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260928 21:17:08.122351 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a73b47ff86d429a9a0d025d0d159085" candidate_uuid: "df07a9f02eb74044b975811ec6b9e82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3f443713a040cd8d4ca684dc13be23" is_pre_election: true
I20260928 21:17:08.122352 9436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a73b47ff86d429a9a0d025d0d159085" candidate_uuid: "df07a9f02eb74044b975811ec6b9e82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f330437544a65b02fbf3b95dcff44" is_pre_election: true
I20260928 21:17:08.122645 9569 raft_consensus.cc:2454] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df07a9f02eb74044b975811ec6b9e82f in term 0.
I20260928 21:17:08.122645 9436 raft_consensus.cc:2454] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df07a9f02eb74044b975811ec6b9e82f in term 0.
I20260928 21:17:08.123098 9638 leader_election.cc:302] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1f330437544a65b02fbf3b95dcff44, df07a9f02eb74044b975811ec6b9e82f; no voters:
I20260928 21:17:08.123325 9772 raft_consensus.cc:2781] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:08.123404 9772 raft_consensus.cc:482] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:08.123451 9772 raft_consensus.cc:3037] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:08.124423 9772 raft_consensus.cc:504] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.124704 9772 leader_election.cc:288] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [CANDIDATE]: Term 1 election: Requested vote from peers cc3f443713a040cd8d4ca684dc13be23 (127.5.18.194:39451), be1f330437544a65b02fbf3b95dcff44 (127.5.18.193:43545)
I20260928 21:17:08.125058 9436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a73b47ff86d429a9a0d025d0d159085" candidate_uuid: "df07a9f02eb74044b975811ec6b9e82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1f330437544a65b02fbf3b95dcff44"
I20260928 21:17:08.125058 9569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a73b47ff86d429a9a0d025d0d159085" candidate_uuid: "df07a9f02eb74044b975811ec6b9e82f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3f443713a040cd8d4ca684dc13be23"
I20260928 21:17:08.125185 9436 raft_consensus.cc:3037] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:08.125191 9569 raft_consensus.cc:3037] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:08.126428 9569 raft_consensus.cc:2454] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df07a9f02eb74044b975811ec6b9e82f in term 1.
I20260928 21:17:08.126428 9436 raft_consensus.cc:2454] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df07a9f02eb74044b975811ec6b9e82f in term 1.
I20260928 21:17:08.126699 9638 leader_election.cc:302] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1f330437544a65b02fbf3b95dcff44, df07a9f02eb74044b975811ec6b9e82f; no voters:
I20260928 21:17:08.126834 9772 raft_consensus.cc:2781] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:08.127091 9772 raft_consensus.cc:686] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 1 LEADER]: Becoming Leader. State: Replica: df07a9f02eb74044b975811ec6b9e82f, State: Running, Role: LEADER
I20260928 21:17:08.127425 9772 consensus_queue.cc:237] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } }
I20260928 21:17:08.130316 9290 catalog_manager.cc:5495] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f reported cstate change: term changed from 0 to 1, leader changed from <none> to df07a9f02eb74044b975811ec6b9e82f (127.5.18.195). New cstate: current_term: 1 leader_uuid: "df07a9f02eb74044b975811ec6b9e82f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df07a9f02eb74044b975811ec6b9e82f" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 32977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 } health_report { overall_health: UNKNOWN } } }
I20260928 21:17:08.144656 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.141224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.5.18.195:32977)
0928 21:17:08.141339 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.141342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.141355 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:08.141479 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.141481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.141537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.141659 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:08.141891 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.142457 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.143268 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:08.143485 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:08.143568 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:08.143867 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:08.144271 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:08.144376 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.144451 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.144504 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:08.150156 9436 raft_consensus.cc:1261] T 8a73b47ff86d429a9a0d025d0d159085 P be1f330437544a65b02fbf3b95dcff44 [term 1 FOLLOWER]: Refusing update from remote peer df07a9f02eb74044b975811ec6b9e82f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:08.150156 9569 raft_consensus.cc:1261] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 1 FOLLOWER]: Refusing update from remote peer df07a9f02eb74044b975811ec6b9e82f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:08.150794 9772 consensus_queue.cc:1035] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3f443713a040cd8d4ca684dc13be23" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:08.151016 9775 consensus_queue.cc:1035] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1f330437544a65b02fbf3b95dcff44" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:08.157784 9778 mvcc.cc:205] Tried to move back new op lower bound from 7334421414496645120 to 7334421414412361728. Current Snapshot: MvccSnapshot[applied={T|T < 7334421414496645120}]
I20260928 21:17:08.166759 9291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:08.166903 9291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:08.168952 9291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 723796c7ef2048d380b3066948f403c2: Sending DeleteTablet for 3 replicas of tablet 8a73b47ff86d429a9a0d025d0d159085
I20260928 21:17:08.169683 9682 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:08 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:55868
I20260928 21:17:08.170116 9787 tablet_replica.cc:330] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: stopping tablet replica
I20260928 21:17:08.170146 9416 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:08 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:37554
I20260928 21:17:08.170395 9787 raft_consensus.cc:2229] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:08.170406 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9354
I20260928 21:17:08.170717 9787 raft_consensus.cc:2258] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:08.171283 9549 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a73b47ff86d429a9a0d025d0d159085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:08 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:55540
I20260928 21:17:08.172281 9787 ts_tablet_manager.cc:1905] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:08.174744 9787 ts_tablet_manager.cc:1918] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:08.174825 9787 log.cc:1196] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/8a73b47ff86d429a9a0d025d0d159085
I20260928 21:17:08.175060 9787 ts_tablet_manager.cc:1939] T 8a73b47ff86d429a9a0d025d0d159085 P df07a9f02eb74044b975811ec6b9e82f: Deleting consensus metadata
I20260928 21:17:08.175992 9277 catalog_manager.cc:4841] TS df07a9f02eb74044b975811ec6b9e82f (127.5.18.195:32977): tablet 8a73b47ff86d429a9a0d025d0d159085 (table test-table [id=c8cf827aa93247f1af996912af1fdebe]) successfully deleted
W20260928 21:17:08.179277 9279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:43545 (error 108)
I20260928 21:17:08.179448 9789 tablet_replica.cc:330] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23: stopping tablet replica
I20260928 21:17:08.179764 9789 raft_consensus.cc:2229] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:08.180084 9789 raft_consensus.cc:2258] T 8a73b47ff86d429a9a0d025d0d159085 P cc3f443713a040cd8d4ca684dc13be23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:08.180233 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.179976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:43545 (local address 127.0.0.1:37562)
0928 21:17:08.180087 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.180150 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:43545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:17:08.180281 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9487
W20260928 21:17:08.181900 9279 catalog_manager.cc:4584] TS be1f330437544a65b02fbf3b95dcff44 (127.5.18.193:43545): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a73b47ff86d429a9a0d025d0d159085: Network error: Client connection negotiation failed: client connection to 127.5.18.193:43545: connect: Connection refused (error 111)
I20260928 21:17:08.186635 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9620
I20260928 21:17:08.188148 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.187892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:39451 (local address 127.0.0.1:55556)
0928 21:17:08.188017 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.188071 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:39451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:17:08.191184 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9261
2026-09-28T21:17:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2101 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:08Z Disabled control of system clock
I20260928 21:17:08.218837 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46425
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:17:08.322635 9794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:08.322891 9794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:08.322925 9794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:08.326793 9794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:08.326853 9794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:08.326866 9794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:08.326881 9794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:08.326894 9794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:08.331532 9794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:08.332548 9794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:08.333953 9794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:08.342242 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.342208 9799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.342322 9794 server_base.cc:1034] running on GCE node
W20260928 21:17:08.342233 9800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.343014 9794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:08.344079 9794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:08.345224 9794 hybrid_clock.cc:648] HybridClock initialized: now 1790630228345200 us; error 43 us; skew 500 ppm
I20260928 21:17:08.346992 9794 webserver.cc:460] Webserver started at http://127.5.18.254:46799/ using document root <none> and password file <none>
I20260928 21:17:08.347536 9794 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:08.347599 9794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:08.347787 9794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:08.349835 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "b3021913e46c40a29db03b78ad7c353c"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "08863dc792f358731ea660da307db1fc"
server_key_iv: "bacb2525a360b4695a8f7f75f4dc6f77"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.350363 9794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "b3021913e46c40a29db03b78ad7c353c"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "08863dc792f358731ea660da307db1fc"
server_key_iv: "bacb2525a360b4695a8f7f75f4dc6f77"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.353828 9794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:17:08.356053 9808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.357219 9794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:08.357327 9794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "b3021913e46c40a29db03b78ad7c353c"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "08863dc792f358731ea660da307db1fc"
server_key_iv: "bacb2525a360b4695a8f7f75f4dc6f77"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.357417 9794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:08.361765 9794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:08.362346 9794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:08.362495 9794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:08.368825 9794 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46425
I20260928 21:17:08.368872 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46425 every 8 connection(s)
I20260928 21:17:08.369732 9794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:08.372746 9861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.373468 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9794
I20260928 21:17:08.373581 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:08.373831 5195 external_mini_cluster.cc:1428] Setting key 22ac17edb8d97259348c4af01a579bd6
I20260928 21:17:08.378403 9861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: Bootstrap starting.
I20260928 21:17:08.378971 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.375124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.5.18.254:46425)
0928 21:17:08.375686 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.375700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.375734 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:08.376351 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.376366 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.376705 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.376974 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.377757 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.377761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.377773 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.377782 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.377834 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.377850 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.377956 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.378049 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.378288 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260928 21:17:08.380646 9861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.381377 9861 log.cc:824] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.383366 9861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: No bootstrap required, opened a new log
I20260928 21:17:08.386772 9861 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } }
I20260928 21:17:08.386981 9861 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.387051 9861 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3021913e46c40a29db03b78ad7c353c, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.387422 9861 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } }
I20260928 21:17:08.387543 9861 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:08.387602 9861 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:08.387696 9861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:08.388505 9861 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } }
I20260928 21:17:08.388805 9861 leader_election.cc:302] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3021913e46c40a29db03b78ad7c353c; no voters:
I20260928 21:17:08.389070 9861 leader_election.cc:288] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:08.389180 9866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:08.389377 9866 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [term 1 LEADER]: Becoming Leader. State: Replica: b3021913e46c40a29db03b78ad7c353c, State: Running, Role: LEADER
I20260928 21:17:08.389672 9866 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } }
I20260928 21:17:08.390192 9861 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:08.391139 9868 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3021913e46c40a29db03b78ad7c353c. Latest consensus state: current_term: 1 leader_uuid: "b3021913e46c40a29db03b78ad7c353c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } } }
I20260928 21:17:08.391254 9868 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:08.391516 9867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3021913e46c40a29db03b78ad7c353c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3021913e46c40a29db03b78ad7c353c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46425 } } }
I20260928 21:17:08.391604 9867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:08.391721 9875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:08.394510 9875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:08.399508 9875 catalog_manager.cc:1347] Generated new cluster ID: 4c0c50033ac04ded92a1d7632dfb6164
I20260928 21:17:08.399557 9875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:08.423785 9875 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:08.424122 9875 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:08.432571 9875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: Generated new TSK 0
I20260928 21:17:08.433341 9875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:08.438092 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--builtin_ntp_servers=127.5.18.212:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:17:08.543427 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:08.543696 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:08.543752 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:08.547616 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:08.547662 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:08.547735 9885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:08.552320 9885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:08.553275 9885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:08.554611 9885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:08.563434 9893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.563463 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.563443 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.563689 9885 server_base.cc:1034] running on GCE node
I20260928 21:17:08.564043 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:08.564623 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:08.565761 9885 hybrid_clock.cc:648] HybridClock initialized: now 1790630228565726 us; error 51 us; skew 500 ppm
I20260928 21:17:08.567430 9885 webserver.cc:460] Webserver started at http://127.5.18.193:34597/ using document root <none> and password file <none>
I20260928 21:17:08.567943 9885 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:08.568005 9885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:08.568187 9885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:08.570465 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "55532afa988544c7bd55485decbd74cf"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "829e7234bc087ac3ee9e8ff6d8a87d3d"
server_key_iv: "f8879af9c1680458110538e2b34d78a5"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.571065 9885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "55532afa988544c7bd55485decbd74cf"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "829e7234bc087ac3ee9e8ff6d8a87d3d"
server_key_iv: "f8879af9c1680458110538e2b34d78a5"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.574724 9885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:08.576915 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.578130 9885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:08.578219 9885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "55532afa988544c7bd55485decbd74cf"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "829e7234bc087ac3ee9e8ff6d8a87d3d"
server_key_iv: "f8879af9c1680458110538e2b34d78a5"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.578302 9885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:08.582402 9885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:08.582914 9885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:08.583036 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:08.583478 9885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:08.584295 9885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:08.584339 9885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.584372 9885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:08.584393 9885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.592599 9885 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:34305
I20260928 21:17:08.592624 10012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:34305 every 8 connection(s)
I20260928 21:17:08.593400 9885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:08.594336 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 9885
I20260928 21:17:08.594429 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:08.594681 5195 external_mini_cluster.cc:1428] Setting key a8b4581e962250e9c4b4a5dcf2825717
I20260928 21:17:08.596567 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--builtin_ntp_servers=127.5.18.212:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:17:08.599192 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.595086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:53875 (local address 127.5.18.254:46425)
0928 21:17:08.595203 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.595206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.596341 (+ 1135us) server_negotiation.cc:407] Connection header received
0928 21:17:08.597552 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.597556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.597621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.597706 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.598821 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.598824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.598826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.598836 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.598855 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.598863 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.598931 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.599055 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.599095 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:17:08.600201 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.595535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46425 (local address 127.5.18.193:53875)
0928 21:17:08.596196 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.596243 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.597340 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.597885 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.597894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.598018 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.598546 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.598548 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.598598 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.598680 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.598934 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.598940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.598945 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.599089 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.599307 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260928 21:17:08.601291 10013 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46425
I20260928 21:17:08.601526 10013 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:08.601984 10013 heartbeater.cc:511] Master 127.5.18.254:46425 requested a full tablet report, sending...
I20260928 21:17:08.603619 9824 ts_manager.cc:194] Registered new tserver with Master: 55532afa988544c7bd55485decbd74cf (127.5.18.193:34305)
W20260928 21:17:08.704867 10016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:08.705113 10016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:08.705148 10016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:08.709018 10016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:08.709072 10016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:08.709126 10016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:08.713689 10016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:08.714676 10016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:08.715969 10016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:08.724641 10021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.724663 10022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.724718 10024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.724854 10016 server_base.cc:1034] running on GCE node
I20260928 21:17:08.725237 10016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:08.725852 10016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:08.727000 10016 hybrid_clock.cc:648] HybridClock initialized: now 1790630228726974 us; error 37 us; skew 500 ppm
I20260928 21:17:08.728691 10016 webserver.cc:460] Webserver started at http://127.5.18.194:33239/ using document root <none> and password file <none>
I20260928 21:17:08.729223 10016 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:08.729286 10016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:08.729460 10016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:08.731480 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "b9dc800aca76437498699eb22a3c144a"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "6f1d5f07c6e0d730f2040db84497adc6"
server_key_iv: "e7e727cb6e0fce07ea04ac56ee5c60b3"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.732013 10016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "b9dc800aca76437498699eb22a3c144a"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "6f1d5f07c6e0d730f2040db84497adc6"
server_key_iv: "e7e727cb6e0fce07ea04ac56ee5c60b3"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.735672 10016 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:08.737957 10030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.739078 10016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:08.739176 10016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "b9dc800aca76437498699eb22a3c144a"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "6f1d5f07c6e0d730f2040db84497adc6"
server_key_iv: "e7e727cb6e0fce07ea04ac56ee5c60b3"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.739264 10016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:08.743530 10016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:08.744077 10016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:08.744221 10016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:08.744701 10016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:08.745523 10016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:08.745586 10016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.745637 10016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:08.745662 10016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.753345 10016 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:41245
I20260928 21:17:08.753391 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:41245 every 8 connection(s)
I20260928 21:17:08.754184 10016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:08.759609 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.755775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:55995 (local address 127.5.18.254:46425)
0928 21:17:08.755900 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.755904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.756936 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 21:17:08.758220 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.758222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.758258 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.758326 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.759263 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.759265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.759267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.759277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.759295 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.759303 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.759368 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.759473 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.759505 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:17:08.760526 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.756144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46425 (local address 127.5.18.194:55995)
0928 21:17:08.756792 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.756831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.758046 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.758470 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.758477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.758560 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.759032 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.759033 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.759072 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.759136 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.759370 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.759374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.759379 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.759514 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.759700 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":108,"threads_started":1}
I20260928 21:17:08.761545 10144 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46425
I20260928 21:17:08.761755 10144 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:08.761950 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10016
I20260928 21:17:08.762050 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:08.762161 10144 heartbeater.cc:511] Master 127.5.18.254:46425 requested a full tablet report, sending...
I20260928 21:17:08.762301 5195 external_mini_cluster.cc:1428] Setting key 4537752deccafd1ad82e27926ebd87ec
I20260928 21:17:08.763130 9825 ts_manager.cc:194] Registered new tserver with Master: b9dc800aca76437498699eb22a3c144a (127.5.18.194:41245)
I20260928 21:17:08.764060 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--builtin_ntp_servers=127.5.18.212:35755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:17:08.870222 10147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:08.870501 10147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:08.870565 10147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:08.874668 10147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:08.874725 10147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:08.874811 10147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:08.879500 10147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:08.880528 10147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:08.881963 10147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:08.891136 10152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:08.891151 10153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.891202 10147 server_base.cc:1034] running on GCE node
W20260928 21:17:08.891134 10155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:08.891703 10147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:08.892325 10147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:08.893482 10147 hybrid_clock.cc:648] HybridClock initialized: now 1790630228893466 us; error 30 us; skew 500 ppm
I20260928 21:17:08.895320 10147 webserver.cc:460] Webserver started at http://127.5.18.195:44385/ using document root <none> and password file <none>
I20260928 21:17:08.895824 10147 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:08.895892 10147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:08.896067 10147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:08.898068 10147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "f90144a47b2945649de64f1148230123"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "3ec4271b9fee3d613e3a0b802124228c"
server_key_iv: "07b5408ff015f347a718f22cd626382d"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.898569 10147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "f90144a47b2945649de64f1148230123"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "3ec4271b9fee3d613e3a0b802124228c"
server_key_iv: "07b5408ff015f347a718f22cd626382d"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.902066 10147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:08.904290 10161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.905418 10147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:08.905531 10147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "f90144a47b2945649de64f1148230123"
format_stamp: "Formatted at 2026-09-28 21:17:08 on dist-test-slave-rc9f"
server_key: "3ec4271b9fee3d613e3a0b802124228c"
server_key_iv: "07b5408ff015f347a718f22cd626382d"
server_key_version: "encryptionkey@0"
I20260928 21:17:08.905654 10147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:08.909821 10147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:08.910348 10147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:08.910486 10147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:08.910972 10147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:08.911784 10147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:08.911827 10147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.911865 10147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:08.911893 10147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:08.919422 10147 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:37317
I20260928 21:17:08.919447 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:37317 every 8 connection(s)
I20260928 21:17:08.920208 10147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:08.925483 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.921811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:41871 (local address 127.5.18.254:46425)
0928 21:17:08.921964 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.921970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.922978 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 21:17:08.924159 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.924161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.924197 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.924264 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.925149 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.925152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.925154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.925165 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.925182 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.925189 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.925258 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.925351 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.925385 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:17:08.926286 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.922219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46425 (local address 127.5.18.195:41871)
0928 21:17:08.922840 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.922877 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.923993 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.924388 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.924394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.924481 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.924929 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.924930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.924967 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.925028 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.925253 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.925258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.925262 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.925376 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.925597 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260928 21:17:08.927258 10275 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46425
I20260928 21:17:08.927455 10275 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:08.927824 10275 heartbeater.cc:511] Master 127.5.18.254:46425 requested a full tablet report, sending...
I20260928 21:17:08.928738 9825 ts_manager.cc:194] Registered new tserver with Master: f90144a47b2945649de64f1148230123 (127.5.18.195:37317)
I20260928 21:17:08.929306 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10147
I20260928 21:17:08.929410 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:08.929675 5195 external_mini_cluster.cc:1428] Setting key 14ee0d31b5c4174b141021aa0b0e08a6
I20260928 21:17:08.931449 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:17:08.934684 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.933170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.5.18.254:46425)
0928 21:17:08.933276 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.933281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.933444 (+ 163us) server_negotiation.cc:407] Connection header received
0928 21:17:08.933557 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.933559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.933617 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.933678 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.934338 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.934341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.934344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.934355 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.934375 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.934383 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.934452 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.934533 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.934571 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:08.937258 9825 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:08.940234 9825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:08.954306 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.951200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34305 (local address 127.0.0.1:36644)
0928 21:17:08.951737 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.951791 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.951892 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.952516 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.952523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.952600 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.953131 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.953133 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.953153 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.953217 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.954055 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.954064 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.954070 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.954148 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.954198 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260928 21:17:08.954782 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.951294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.5.18.193:34305)
0928 21:17:08.951602 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.951612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.951863 (+ 251us) server_negotiation.cc:407] Connection header received
0928 21:17:08.951995 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.952003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.952331 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.952867 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.953856 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.953862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.953873 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.953883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.953922 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.953951 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.954488 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.954547 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.954619 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
I20260928 21:17:08.954900 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.951923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:41245 (local address 127.0.0.1:53622)
0928 21:17:08.952162 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.952173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.952243 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.953340 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.953343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.953357 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.953959 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.953961 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.953972 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.953998 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.954667 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.954670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.954673 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.954730 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.954765 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260928 21:17:08.954905 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.952240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37317 (local address 127.0.0.1:54814)
0928 21:17:08.952420 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:08.952429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:08.952476 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:08.953558 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:08.953561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:08.953605 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:08.953964 (+ 359us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:08.953966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:08.953976 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:08.953999 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:08.954742 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:08.954746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:08.954748 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 21:17:08.954791 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 21:17:08.954814 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 21:17:08.955487 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.952620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.5.18.194:41245)
0928 21:17:08.952951 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.952962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.952987 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:17:08.953037 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.953042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.953217 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.953344 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.954148 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.954154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.954165 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.954177 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.954218 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.954249 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.955209 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.955258 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.955324 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
I20260928 21:17:08.955487 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:08.952748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.5.18.195:37317)
0928 21:17:08.953163 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:08.953175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:08.953198 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:17:08.953247 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:08.953252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:08.953413 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:08.953784 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:08.954379 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:08.954384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:08.954394 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:08.954403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:08.954618 (+ 215us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:08.954656 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:08.955248 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:08.955286 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 21:17:08.955336 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":132,"threads_started":1}
I20260928 21:17:08.957412 9947 tablet_service.cc:1467] Processing CreateTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f (DEFAULT_TABLE table=test-table [id=43d660a2b57d42c4be4e75bed7f75607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.958451 9947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fcdc6c81d44171927e5b3a39ea137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.958222 10209 tablet_service.cc:1467] Processing CreateTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f (DEFAULT_TABLE table=test-table [id=43d660a2b57d42c4be4e75bed7f75607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.958222 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f (DEFAULT_TABLE table=test-table [id=43d660a2b57d42c4be4e75bed7f75607]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:08.959199 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fcdc6c81d44171927e5b3a39ea137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.959199 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74fcdc6c81d44171927e5b3a39ea137f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:08.965198 10292 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Bootstrap starting.
I20260928 21:17:08.966395 10294 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Bootstrap starting.
I20260928 21:17:08.967571 10292 tablet_bootstrap.cc:654] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.968022 10294 tablet_bootstrap.cc:654] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.968283 10292 log.cc:824] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.968650 10294 log.cc:824] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.969239 10293 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Bootstrap starting.
I20260928 21:17:08.970247 10292 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: No bootstrap required, opened a new log
I20260928 21:17:08.970419 10292 ts_tablet_manager.cc:1397] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:17:08.970757 10294 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: No bootstrap required, opened a new log
I20260928 21:17:08.970930 10294 ts_tablet_manager.cc:1397] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:17:08.971218 10293 tablet_bootstrap.cc:654] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:08.971982 10293 log.cc:824] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:08.973961 10293 tablet_bootstrap.cc:492] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: No bootstrap required, opened a new log
I20260928 21:17:08.973887 10292 raft_consensus.cc:348] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.974133 10293 ts_tablet_manager.cc:1397] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:17:08.974136 10292 raft_consensus.cc:374] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.974184 10292 raft_consensus.cc:729] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55532afa988544c7bd55485decbd74cf, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.974609 10292 consensus_queue.cc:260] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.975010 10294 raft_consensus.cc:348] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.975258 10294 raft_consensus.cc:374] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.975313 10294 raft_consensus.cc:729] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9dc800aca76437498699eb22a3c144a, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.975701 10292 ts_tablet_manager.cc:1428] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:17:08.975705 10294 consensus_queue.cc:260] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.976025 10013 heartbeater.cc:503] Master 127.5.18.254:46425 was elected leader, sending a full tablet report...
I20260928 21:17:08.976604 10294 ts_tablet_manager.cc:1428] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 21:17:08.976923 10144 heartbeater.cc:503] Master 127.5.18.254:46425 was elected leader, sending a full tablet report...
I20260928 21:17:08.978118 10293 raft_consensus.cc:348] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.978340 10293 raft_consensus.cc:374] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:08.978396 10293 raft_consensus.cc:729] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90144a47b2945649de64f1148230123, State: Initialized, Role: FOLLOWER
I20260928 21:17:08.978753 10293 consensus_queue.cc:260] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:08.979482 10275 heartbeater.cc:503] Master 127.5.18.254:46425 was elected leader, sending a full tablet report...
I20260928 21:17:08.979610 10293 ts_tablet_manager.cc:1428] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 21:17:09.005023 10145 tablet.cc:2369] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:17:09.094448 10014 tablet.cc:2369] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:09.169142 10298 raft_consensus.cc:482] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:09.169373 10298 raft_consensus.cc:504] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:09.170579 10298 leader_election.cc:288] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9dc800aca76437498699eb22a3c144a (127.5.18.194:41245), f90144a47b2945649de64f1148230123 (127.5.18.195:37317)
W20260928 21:17:09.171072 10276 tablet.cc:2369] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:09.172595 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.170995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:47021 (local address 127.5.18.194:41245)
0928 21:17:09.171132 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.171137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.171156 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:17:09.171321 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.171325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.171368 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.171422 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:09.172207 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:09.172211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:09.172214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:09.172227 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:09.172249 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:09.172260 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:09.172347 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:09.172431 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:17:09.172472 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:17:09.172605 10301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.170823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:41245 (local address 127.5.18.193:47021)
0928 21:17:09.171092 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:09.171107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:09.171193 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:09.171543 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:09.171545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:09.171560 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:09.172077 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:09.172079 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:09.172088 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:09.172111 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:09.172338 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:09.172341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:09.172344 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:09.172431 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:17:09.172465 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260928 21:17:09.172878 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.171331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:42641 (local address 127.5.18.195:37317)
0928 21:17:09.171445 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.171449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.171706 (+ 257us) server_negotiation.cc:407] Connection header received
0928 21:17:09.171786 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.171788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.171819 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.171877 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:09.172468 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:09.172471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:09.172474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:09.172485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:09.172505 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:09.172514 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:09.172595 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:09.172728 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:17:09.172769 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:17:09.173177 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.171198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37317 (local address 127.5.18.193:42641)
0928 21:17:09.171585 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:09.171598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:09.171710 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:09.171916 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:09.171920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:09.171935 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:09.172327 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:09.172329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:09.172338 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:09.172359 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:09.172607 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:09.172609 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:09.172612 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:09.173015 (+ 403us) client_negotiation.cc:240] Negotiation successful
0928 21:17:09.173050 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":63,"threads_started":1}
I20260928 21:17:09.173192 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fcdc6c81d44171927e5b3a39ea137f" candidate_uuid: "55532afa988544c7bd55485decbd74cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc800aca76437498699eb22a3c144a" is_pre_election: true
I20260928 21:17:09.173465 10098 raft_consensus.cc:2454] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55532afa988544c7bd55485decbd74cf in term 0.
I20260928 21:17:09.173834 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fcdc6c81d44171927e5b3a39ea137f" candidate_uuid: "55532afa988544c7bd55485decbd74cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90144a47b2945649de64f1148230123" is_pre_election: true
I20260928 21:17:09.173950 9900 leader_election.cc:302] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55532afa988544c7bd55485decbd74cf, b9dc800aca76437498699eb22a3c144a; no voters:
I20260928 21:17:09.174208 10229 raft_consensus.cc:2454] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55532afa988544c7bd55485decbd74cf in term 0.
I20260928 21:17:09.174301 10298 raft_consensus.cc:2781] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:09.174396 10298 raft_consensus.cc:482] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:09.174443 10298 raft_consensus.cc:3037] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:09.175432 10298 raft_consensus.cc:504] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:09.175734 10298 leader_election.cc:288] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [CANDIDATE]: Term 1 election: Requested vote from peers b9dc800aca76437498699eb22a3c144a (127.5.18.194:41245), f90144a47b2945649de64f1148230123 (127.5.18.195:37317)
I20260928 21:17:09.176057 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fcdc6c81d44171927e5b3a39ea137f" candidate_uuid: "55532afa988544c7bd55485decbd74cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc800aca76437498699eb22a3c144a"
I20260928 21:17:09.176093 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74fcdc6c81d44171927e5b3a39ea137f" candidate_uuid: "55532afa988544c7bd55485decbd74cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f90144a47b2945649de64f1148230123"
I20260928 21:17:09.176183 10229 raft_consensus.cc:3037] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:09.176182 10098 raft_consensus.cc:3037] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:09.177393 10098 raft_consensus.cc:2454] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55532afa988544c7bd55485decbd74cf in term 1.
I20260928 21:17:09.177393 10229 raft_consensus.cc:2454] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55532afa988544c7bd55485decbd74cf in term 1.
I20260928 21:17:09.177686 9900 leader_election.cc:302] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55532afa988544c7bd55485decbd74cf, b9dc800aca76437498699eb22a3c144a; no voters:
I20260928 21:17:09.177834 10298 raft_consensus.cc:2781] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:09.178021 10298 raft_consensus.cc:686] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 1 LEADER]: Becoming Leader. State: Replica: 55532afa988544c7bd55485decbd74cf, State: Running, Role: LEADER
I20260928 21:17:09.178330 10298 consensus_queue.cc:237] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } }
I20260928 21:17:09.181322 9825 catalog_manager.cc:5495] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 55532afa988544c7bd55485decbd74cf (127.5.18.193). New cstate: current_term: 1 leader_uuid: "55532afa988544c7bd55485decbd74cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55532afa988544c7bd55485decbd74cf" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 } health_report { overall_health: UNKNOWN } } }
I20260928 21:17:09.247577 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.246230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.5.18.193:34305)
0928 21:17:09.246349 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.246353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.246366 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:09.246443 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.246445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.246505 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.246584 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:09.247291 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:09.247293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:09.247295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:09.247302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:09.247315 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:09.247321 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:09.247379 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:09.247432 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:17:09.247463 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:17:09.253667 10098 raft_consensus.cc:1261] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 1 FOLLOWER]: Refusing update from remote peer 55532afa988544c7bd55485decbd74cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:09.253669 10229 raft_consensus.cc:1261] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 1 FOLLOWER]: Refusing update from remote peer 55532afa988544c7bd55485decbd74cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:09.254366 10303 consensus_queue.cc:1035] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9dc800aca76437498699eb22a3c144a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 41245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:09.254542 10298 consensus_queue.cc:1035] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90144a47b2945649de64f1148230123" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:09.263149 10308 mvcc.cc:205] Tried to move back new op lower bound from 7334421419015639040 to 7334421418716971008. Current Snapshot: MvccSnapshot[applied={T|T < 7334421419015639040}]
I20260928 21:17:09.269435 9825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:09.269623 9825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:09.271965 9825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3021913e46c40a29db03b78ad7c353c: Sending DeleteTablet for 3 replicas of tablet 74fcdc6c81d44171927e5b3a39ea137f
I20260928 21:17:09.272631 9947 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:09 UTC) from {username='slave'} at 127.0.0.1:36644
I20260928 21:17:09.272682 10078 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:09 UTC) from {username='slave'} at 127.0.0.1:53622
I20260928 21:17:09.272742 10209 tablet_service.cc:1514] Processing DeleteTablet for tablet 74fcdc6c81d44171927e5b3a39ea137f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:09 UTC) from {username='slave'} at 127.0.0.1:54814
I20260928 21:17:09.273096 10316 tablet_replica.cc:330] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf: stopping tablet replica
I20260928 21:17:09.273416 10316 raft_consensus.cc:2229] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:09.273509 10317 tablet_replica.cc:330] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: stopping tablet replica
I20260928 21:17:09.273820 10317 raft_consensus.cc:2229] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:09.273830 10316 raft_consensus.cc:2258] T 74fcdc6c81d44171927e5b3a39ea137f P 55532afa988544c7bd55485decbd74cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:09.273878 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9885
I20260928 21:17:09.274106 10317 raft_consensus.cc:2258] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:09.275897 10318 tablet_replica.cc:330] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: stopping tablet replica
I20260928 21:17:09.276147 10318 raft_consensus.cc:2229] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:09.276387 10318 raft_consensus.cc:2258] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:09.277664 10318 ts_tablet_manager.cc:1905] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:09.277664 10317 ts_tablet_manager.cc:1905] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:09.279906 10318 ts_tablet_manager.cc:1918] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:09.280025 10318 log.cc:1196] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/74fcdc6c81d44171927e5b3a39ea137f
I20260928 21:17:09.280306 10318 ts_tablet_manager.cc:1939] T 74fcdc6c81d44171927e5b3a39ea137f P f90144a47b2945649de64f1148230123: Deleting consensus metadata
I20260928 21:17:09.280364 10317 ts_tablet_manager.cc:1918] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:09.280447 10317 log.cc:1196] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/74fcdc6c81d44171927e5b3a39ea137f
I20260928 21:17:09.280766 10317 ts_tablet_manager.cc:1939] T 74fcdc6c81d44171927e5b3a39ea137f P b9dc800aca76437498699eb22a3c144a: Deleting consensus metadata
W20260928 21:17:09.280946 9809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:34305 (error 108)
I20260928 21:17:09.281455 9809 catalog_manager.cc:4841] TS f90144a47b2945649de64f1148230123 (127.5.18.195:37317): tablet 74fcdc6c81d44171927e5b3a39ea137f (table test-table [id=43d660a2b57d42c4be4e75bed7f75607]) successfully deleted
I20260928 21:17:09.281883 9809 catalog_manager.cc:4841] TS b9dc800aca76437498699eb22a3c144a (127.5.18.194:41245): tablet 74fcdc6c81d44171927e5b3a39ea137f (table test-table [id=43d660a2b57d42c4be4e75bed7f75607]) successfully deleted
I20260928 21:17:09.282060 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.281765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34305 (local address 127.0.0.1:36652)
0928 21:17:09.281893 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:09.281965 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:17:09.282197 9809 catalog_manager.cc:4584] TS 55532afa988544c7bd55485decbd74cf (127.5.18.193:34305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74fcdc6c81d44171927e5b3a39ea137f: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34305: connect: Connection refused (error 111)
I20260928 21:17:09.282256 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10016
I20260928 21:17:09.287220 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10147
I20260928 21:17:09.291680 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 9794
2026-09-28T21:17:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1098 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:17:09.310994 5195 mini_oidc.cc:152] Starting JWKS server
I20260928 21:17:09.312520 5195 webserver.cc:460] Webserver started at https://127.0.0.1:39387/ using document root <none> and password file <none>
I20260928 21:17:09.312672 5195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:17:09.312927 5195 webserver.cc:460] Webserver started at http://127.0.0.1:39975/ using document root <none> and password file <none>
2026-09-28T21:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:09Z Disabled control of system clock
I20260928 21:17:09.320545 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46873
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39387/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/testchainca.cert with env {}
W20260928 21:17:09.424614 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:09.424865 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:09.424909 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:09.428858 10325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:09.428915 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:09.428929 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:09.428943 10325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:09.428966 10325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:17:09.428985 10325 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:17:09.429001 10325 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39387/jwks.json
I20260928 21:17:09.434142 10325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39387/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:09.435130 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:09.436036 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:09.443796 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:09.443802 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:09.443802 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:09.443836 10325 server_base.cc:1034] running on GCE node
I20260928 21:17:09.444491 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:09.445479 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:09.446641 10325 hybrid_clock.cc:648] HybridClock initialized: now 1790630229446627 us; error 30 us; skew 500 ppm
I20260928 21:17:09.448428 10325 webserver.cc:460] Webserver started at http://127.5.18.254:39051/ using document root <none> and password file <none>
I20260928 21:17:09.448971 10325 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:09.449048 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:09.449256 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:09.451277 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "99bd95ea015b4d72a441472848eb139d"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "0a15a6b60ad1c56a0292a753a72da951"
server_key_iv: "f27376de86befb7cf43f9b0c277d2468"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.451849 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "99bd95ea015b4d72a441472848eb139d"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "0a15a6b60ad1c56a0292a753a72da951"
server_key_iv: "f27376de86befb7cf43f9b0c277d2468"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.455495 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:09.457862 10339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:09.459019 10325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:17:09.459144 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "99bd95ea015b4d72a441472848eb139d"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "0a15a6b60ad1c56a0292a753a72da951"
server_key_iv: "f27376de86befb7cf43f9b0c277d2468"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.459236 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:09.470120 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:09.470705 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:09.470857 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:09.477123 10325 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46873
I20260928 21:17:09.477142 10391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46873 every 8 connection(s)
I20260928 21:17:09.477984 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:09.480593 10392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:09.485002 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10325
I20260928 21:17:09.485100 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:09.485324 5195 external_mini_cluster.cc:1428] Setting key 203f8c9c20fbef4028b88d798d07837b
I20260928 21:17:09.485913 10392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: Bootstrap starting.
I20260928 21:17:09.488395 10392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:09.489233 10392 log.cc:824] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:09.491184 10392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: No bootstrap required, opened a new log
I20260928 21:17:09.495256 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.486564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.5.18.254:46873)
0928 21:17:09.487239 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.487257 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.487298 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:17:09.488074 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.488097 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.488584 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.488935 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:09.489386 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.490705 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.491616 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.492299 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.492459 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.493195 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:09.493201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:09.493217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:09.493713 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:09.493780 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:09.493805 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:09.493899 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:09.494028 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 21:17:09.494313 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260928 21:17:09.495424 10392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } }
I20260928 21:17:09.495668 10392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:09.495728 10392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bd95ea015b4d72a441472848eb139d, State: Initialized, Role: FOLLOWER
I20260928 21:17:09.496129 10392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } }
I20260928 21:17:09.496264 10392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:09.496326 10392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:09.496411 10392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:09.497399 10392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } }
I20260928 21:17:09.497781 10392 leader_election.cc:302] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99bd95ea015b4d72a441472848eb139d; no voters:
I20260928 21:17:09.498088 10392 leader_election.cc:288] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:09.498217 10397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:09.498248 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:17:09.498445 10397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [term 1 LEADER]: Becoming Leader. State: Replica: 99bd95ea015b4d72a441472848eb139d, State: Running, Role: LEADER
I20260928 21:17:09.498744 10397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } }
I20260928 21:17:09.499259 10392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:09.500429 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99bd95ea015b4d72a441472848eb139d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } } }
I20260928 21:17:09.500536 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:09.500766 10398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99bd95ea015b4d72a441472848eb139d. Latest consensus state: current_term: 1 leader_uuid: "99bd95ea015b4d72a441472848eb139d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd95ea015b4d72a441472848eb139d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46873 } } }
I20260928 21:17:09.500892 10398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d [sys.catalog]: This master's current role is: LEADER
W20260928 21:17:09.503734 10412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:17:09.503799 10412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:17:09.503865 10407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:09.504518 10407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:09.509188 10407 catalog_manager.cc:1347] Generated new cluster ID: 907f1664b6b544ac884afacb0aea503a
I20260928 21:17:09.509244 10407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:09.516281 10407 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:09.517140 10407 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:09.521647 10407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99bd95ea015b4d72a441472848eb139d: Generated new TSK 0
I20260928 21:17:09.522212 10407 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:17:09.554275 10422 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:17:09.554385 10422 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:17:09.554845 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.531013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.5.18.254:46873)
0928 21:17:09.531131 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.531136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.531327 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:17:09.531520 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.531523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.531566 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.531653 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:17:09.531883 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.532398 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.533664 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.533892 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.533975 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.534291 (+ 316us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:17:09.554418 (+ 20127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:17:09.554694 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:17:09.554970 10395 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:17:09.556012 10421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:09.530926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46873 (local address 127.0.0.1:47588)
0928 21:17:09.531248 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:09.531263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:09.531391 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:09.531675 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:09.531678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:09.531685 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:17:09.531793 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:09.531799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.532507 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.532511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:09.533510 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:09.533521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.533997 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.534001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:09.534092 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.534141 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:17:09.554644 (+ 20503us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:17:09.554881 (+ 237us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.18.254:46873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.18.254:46873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39387/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:17:09.556627 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10325
2026-09-28T21:17:09Z chronyd exiting
I20260928 21:17:09.592818 5195 test_util.cc:175] -----------------------------------------------
I20260928 21:17:09.592900 5195 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790630206151656-5195-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (283 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:17:09.593962 5195 mini_oidc.cc:152] Starting JWKS server
I20260928 21:17:09.594787 5195 webserver.cc:460] Webserver started at https://127.0.0.1:36671/ using document root <none> and password file <none>
I20260928 21:17:09.594882 5195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:17:09.595131 5195 webserver.cc:460] Webserver started at http://127.0.0.1:44025/ using document root <none> and password file <none>
2026-09-28T21:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:09Z Disabled control of system clock
I20260928 21:17:09.602829 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45655
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:45655
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/testchainca.cert with env {}
W20260928 21:17:09.707921 10428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:09.708186 10428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:09.708225 10428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:09.712152 10428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:09.712200 10428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:09.712213 10428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:09.712229 10428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:09.712242 10428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:17:09.712256 10428 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:17:09.712268 10428 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36671/jwks.json
I20260928 21:17:09.717468 10428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:45655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:09.718464 10428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:09.719482 10428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:09.727415 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:09.727522 10428 server_base.cc:1034] running on GCE node
W20260928 21:17:09.727415 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:09.727432 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:09.728186 10428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:09.729216 10428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:09.730360 10428 hybrid_clock.cc:648] HybridClock initialized: now 1790630229730337 us; error 38 us; skew 500 ppm
I20260928 21:17:09.732054 10428 webserver.cc:460] Webserver started at http://127.5.18.254:46009/ using document root <none> and password file <none>
I20260928 21:17:09.732587 10428 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:09.732650 10428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:09.732822 10428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:09.734776 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "f0e14018d2976bc9c3fea25bd2c3331c"
server_key_iv: "2ba9aea6af27eee01bfda9d975d28b57"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.735287 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "f0e14018d2976bc9c3fea25bd2c3331c"
server_key_iv: "2ba9aea6af27eee01bfda9d975d28b57"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.738634 10428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:09.740792 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:09.741950 10428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:09.742055 10428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf"
format_stamp: "Formatted at 2026-09-28 21:17:09 on dist-test-slave-rc9f"
server_key: "f0e14018d2976bc9c3fea25bd2c3331c"
server_key_iv: "2ba9aea6af27eee01bfda9d975d28b57"
server_key_version: "encryptionkey@0"
I20260928 21:17:09.742142 10428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:09.766355 10428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:09.767057 10428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:09.767235 10428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:09.773651 10428 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:45655
I20260928 21:17:09.773715 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:45655 every 8 connection(s)
I20260928 21:17:09.774606 10428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:09.777714 10495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:09.777792 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10428
I20260928 21:17:09.777904 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:09.778172 5195 external_mini_cluster.cc:1428] Setting key dacb6a32f8bd41e3e9d48871f8e91936
I20260928 21:17:09.784082 10495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf: Bootstrap starting.
I20260928 21:17:09.786335 10495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:09.786934 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.779471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.5.18.254:45655)
0928 21:17:09.780107 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.780123 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.780157 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:09.780869 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.780887 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.781280 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.781618 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:09.782027 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.782968 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.783832 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.784305 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.784442 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.785231 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:09.785236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:09.785248 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:09.785663 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:09.785719 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:09.785739 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:09.785820 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:09.785948 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:17:09.786208 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260928 21:17:09.787068 10495 log.cc:824] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:09.789006 10495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf: No bootstrap required, opened a new log
I20260928 21:17:09.789587 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:17:09.792841 10495 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } }
I20260928 21:17:09.793062 10495 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:09.793110 10495 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6aa9748aabf47e190e11c5c6e7b9fdf, State: Initialized, Role: FOLLOWER
I20260928 21:17:09.793485 10495 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } }
I20260928 21:17:09.793624 10495 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:09.793684 10495 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:09.793776 10495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:09.794633 10495 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } }
I20260928 21:17:09.794981 10495 leader_election.cc:302] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6aa9748aabf47e190e11c5c6e7b9fdf; no voters:
I20260928 21:17:09.795270 10495 leader_election.cc:288] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:09.795403 10500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:09.795583 10500 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [term 1 LEADER]: Becoming Leader. State: Replica: c6aa9748aabf47e190e11c5c6e7b9fdf, State: Running, Role: LEADER
I20260928 21:17:09.795869 10500 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } }
I20260928 21:17:09.796434 10495 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:09.797326 10502 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6aa9748aabf47e190e11c5c6e7b9fdf. Latest consensus state: current_term: 1 leader_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } } }
I20260928 21:17:09.797346 10501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6aa9748aabf47e190e11c5c6e7b9fdf" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45655 } } }
I20260928 21:17:09.797461 10501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:09.797461 10502 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:09.797865 10509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:09.800516 10509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:09.805971 10509 catalog_manager.cc:1347] Generated new cluster ID: 2005b580a14a4892bca359d2ae16cf24
I20260928 21:17:09.806039 10509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:09.814255 10509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:09.815464 10509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:09.836212 10509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6aa9748aabf47e190e11c5c6e7b9fdf: Generated new TSK 0
I20260928 21:17:09.837077 10509 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:17:09.866950 10525 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:17:09.867026 10525 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:17:09.867825 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:09.852145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.5.18.254:45655)
0928 21:17:09.852266 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:09.852272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:09.852451 (+ 179us) server_negotiation.cc:407] Connection header received
0928 21:17:09.852588 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:09.852592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:09.852652 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:09.852784 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:17:09.852939 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.853789 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.854600 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.854941 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.855052 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.855280 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:17:09.867365 (+ 12085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:17:09.867671 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 21:17:09.867866 10524 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:09.852038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45655 (local address 127.0.0.1:57312)
0928 21:17:09.852373 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:09.852385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:09.852478 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:09.852768 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:09.852770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:09.852776 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:17:09.852864 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:09.852868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.853892 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.853894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:09.854495 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:09.854503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:09.855042 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:09.855044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:09.855106 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:09.855127 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:17:09.867579 (+ 12452us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:17:09.867729 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.18.254:45655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":104,"threads_started":1}
W20260928 21:17:09.867898 10498 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 21:17:09.868768 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10428
2026-09-28T21:17:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:17:09.906443 5195 mini_oidc.cc:152] Starting JWKS server
I20260928 21:17:09.907256 5195 webserver.cc:460] Webserver started at https://127.0.0.1:39007/ using document root <none> and password file <none>
I20260928 21:17:09.907349 5195 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:17:09.907594 5195 webserver.cc:460] Webserver started at http://127.0.0.1:39487/ using document root <none> and password file <none>
2026-09-28T21:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:09Z Disabled control of system clock
I20260928 21:17:09.915438 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43757
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:43757
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:17:10.019026 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:10.019327 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:10.019371 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:10.023238 10531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:10.023283 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:10.023295 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:10.023310 10531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:10.023324 10531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:17:10.023339 10531 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:17:10.023350 10531 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39007/jwks.json
I20260928 21:17:10.028002 10531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:43757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:10.028946 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:10.030316 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:10.038379 10537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:10.038419 10536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:10.038522 10531 server_base.cc:1034] running on GCE node
W20260928 21:17:10.038403 10539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:10.039072 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:10.040112 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:10.041277 10531 hybrid_clock.cc:648] HybridClock initialized: now 1790630230041257 us; error 30 us; skew 500 ppm
I20260928 21:17:10.043136 10531 webserver.cc:460] Webserver started at http://127.5.18.254:35793/ using document root <none> and password file <none>
I20260928 21:17:10.043682 10531 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:10.043748 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:10.043947 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:10.045912 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "b5c89c9a12054f5caabc58a540a387c1"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "452779f2ab00965c0c0fcf6a7076b67d"
server_key_iv: "f9ab296bc22c50244ac3ab5160f6f1ec"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.046393 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "b5c89c9a12054f5caabc58a540a387c1"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "452779f2ab00965c0c0fcf6a7076b67d"
server_key_iv: "f9ab296bc22c50244ac3ab5160f6f1ec"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.049736 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:10.051908 10545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:10.053053 10531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:10.053148 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "b5c89c9a12054f5caabc58a540a387c1"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "452779f2ab00965c0c0fcf6a7076b67d"
server_key_iv: "f9ab296bc22c50244ac3ab5160f6f1ec"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.053242 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:10.070600 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:10.071241 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:10.071403 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:10.077709 10531 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:43757
I20260928 21:17:10.077716 10597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:43757 every 8 connection(s)
I20260928 21:17:10.078589 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:10.080144 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10531
I20260928 21:17:10.080252 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:10.080515 5195 external_mini_cluster.cc:1428] Setting key 6f0d53d8812abc762625e5405a5c9c57
I20260928 21:17:10.081950 10598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:10.087797 10598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1: Bootstrap starting.
I20260928 21:17:10.089382 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:10.081859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.5.18.254:43757)
0928 21:17:10.082450 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:10.082465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:10.082500 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:17:10.083165 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:10.083181 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:10.083561 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:10.083839 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:10.084167 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.085103 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.086289 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.086746 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.086865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.087723 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:10.087727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:10.087738 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:10.088158 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:10.088213 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:10.088233 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:10.088303 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:10.088449 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:17:10.088698 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
I20260928 21:17:10.090096 10598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:10.090832 10598 log.cc:824] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:10.092092 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:17:10.092857 10598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1: No bootstrap required, opened a new log
I20260928 21:17:10.096102 10598 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } }
I20260928 21:17:10.096303 10598 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:10.096347 10598 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5c89c9a12054f5caabc58a540a387c1, State: Initialized, Role: FOLLOWER
I20260928 21:17:10.096710 10598 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } }
I20260928 21:17:10.096822 10598 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:10.096877 10598 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:10.096971 10598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:10.097813 10598 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } }
I20260928 21:17:10.098107 10598 leader_election.cc:302] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5c89c9a12054f5caabc58a540a387c1; no voters:
I20260928 21:17:10.098387 10598 leader_election.cc:288] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:10.098498 10603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:10.098693 10603 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [term 1 LEADER]: Becoming Leader. State: Replica: b5c89c9a12054f5caabc58a540a387c1, State: Running, Role: LEADER
I20260928 21:17:10.099020 10603 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } }
I20260928 21:17:10.099452 10598 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:10.100497 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5c89c9a12054f5caabc58a540a387c1. Latest consensus state: current_term: 1 leader_uuid: "b5c89c9a12054f5caabc58a540a387c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } } }
I20260928 21:17:10.100629 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:10.100978 10612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:10.100936 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5c89c9a12054f5caabc58a540a387c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5c89c9a12054f5caabc58a540a387c1" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43757 } } }
I20260928 21:17:10.101137 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:10.104121 10612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:10.109323 10612 catalog_manager.cc:1347] Generated new cluster ID: 19f5dc4ff62a46549dd4ffb9fe51833f
I20260928 21:17:10.109375 10612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:10.119050 10612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:10.120213 10612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:10.130511 10612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5c89c9a12054f5caabc58a540a387c1: Generated new TSK 0
I20260928 21:17:10.131325 10612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:10.155493 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:10.150929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.5.18.254:43757)
0928 21:17:10.151053 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:10.151058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:10.151186 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:17:10.151341 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:10.151343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:10.151390 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:10.151487 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:17:10.151734 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.152296 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.153148 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.153366 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.153451 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.153785 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:17:10.155125 (+ 1340us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:17:10.155348 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 21:17:10.155566 10601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 21:17:10.157440 10627 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:10.150821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43757 (local address 127.0.0.1:40654)
0928 21:17:10.151094 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:10.151108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:10.151220 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:10.151492 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:10.151495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:10.151502 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:17:10.151617 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:10.151636 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.152414 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.152418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:10.153036 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:10.153043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.153471 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.153474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:10.153585 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.153613 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:17:10.155295 (+ 1682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:17:10.157264 (+ 1969us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.18.254:43757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39007/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1778,"thread_start_us":99,"threads_started":1}
I20260928 21:17:10.158041 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10531
E20260928 21:17:10.165711 10628 webserver.cc:533] Webserver: sslize failed:
E20260928 21:17:10.165772 10628 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:17:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (289 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:10Z Disabled control of system clock
I20260928 21:17:10.203379 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:41913
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:41913
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/testchainca.cert with env {}
W20260928 21:17:10.307933 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:10.308184 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:10.308223 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:10.312090 10632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:10.312140 10632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:10.312155 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:10.312168 10632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:10.312181 10632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:17:10.312204 10632 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:17:10.312220 10632 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:17:10.317343 10632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:41913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:10.318331 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:10.319229 10632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:10.326908 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:10.326946 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:10.326946 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:10.327016 10632 server_base.cc:1034] running on GCE node
I20260928 21:17:10.327770 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:10.328778 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:10.329927 10632 hybrid_clock.cc:648] HybridClock initialized: now 1790630230329905 us; error 36 us; skew 500 ppm
I20260928 21:17:10.331637 10632 webserver.cc:460] Webserver started at http://127.5.18.254:37501/ using document root <none> and password file <none>
I20260928 21:17:10.332157 10632 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:10.332221 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:10.332405 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:10.334354 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "0427117679114510b12d203f88ed6256"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "56e64541c86c0c4af371abef7eed28ba"
server_key_iv: "f5b8307e327b369e45b52f51b6060a75"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.334859 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "0427117679114510b12d203f88ed6256"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "56e64541c86c0c4af371abef7eed28ba"
server_key_iv: "f5b8307e327b369e45b52f51b6060a75"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.338323 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:10.340607 10646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:10.341825 10632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:10.341931 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "0427117679114510b12d203f88ed6256"
format_stamp: "Formatted at 2026-09-28 21:17:10 on dist-test-slave-rc9f"
server_key: "56e64541c86c0c4af371abef7eed28ba"
server_key_iv: "f5b8307e327b369e45b52f51b6060a75"
server_key_version: "encryptionkey@0"
I20260928 21:17:10.342022 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:10.355316 10632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:10.355950 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:10.356125 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:10.362569 10632 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:41913
I20260928 21:17:10.362586 10698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:41913 every 8 connection(s)
I20260928 21:17:10.363488 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:10.366252 10699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:10.368124 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10632
I20260928 21:17:10.368233 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:10.368475 5195 external_mini_cluster.cc:1428] Setting key 7ccc6f6be2462660d95b81c554c70290
I20260928 21:17:10.373107 10699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256: Bootstrap starting.
I20260928 21:17:10.375569 10699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:10.376384 10699 log.cc:824] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:10.378046 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:10.369787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.5.18.254:41913)
0928 21:17:10.370431 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:10.370451 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:10.370492 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:17:10.371279 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:10.371302 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:10.371815 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:10.372181 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:10.372604 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.374016 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.374828 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.375467 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.375627 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.376347 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:10.376352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:10.376364 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:10.376774 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:10.376841 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:10.376867 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:10.376962 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:10.377093 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:17:10.377339 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260928 21:17:10.378230 10699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256: No bootstrap required, opened a new log
I20260928 21:17:10.380676 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:17:10.382078 10699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } }
I20260928 21:17:10.382324 10699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:10.382388 10699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0427117679114510b12d203f88ed6256, State: Initialized, Role: FOLLOWER
I20260928 21:17:10.382761 10699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } }
I20260928 21:17:10.382871 10699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:10.382936 10699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:10.383028 10699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:10.383782 10699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } }
I20260928 21:17:10.384080 10699 leader_election.cc:302] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0427117679114510b12d203f88ed6256; no voters:
I20260928 21:17:10.384326 10699 leader_election.cc:288] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:10.384413 10704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:10.384626 10704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [term 1 LEADER]: Becoming Leader. State: Replica: 0427117679114510b12d203f88ed6256, State: Running, Role: LEADER
I20260928 21:17:10.384902 10704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } }
I20260928 21:17:10.385381 10699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:10.386457 10706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0427117679114510b12d203f88ed6256. Latest consensus state: current_term: 1 leader_uuid: "0427117679114510b12d203f88ed6256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } } }
I20260928 21:17:10.386585 10706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:10.386994 10705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0427117679114510b12d203f88ed6256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0427117679114510b12d203f88ed6256" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41913 } } }
I20260928 21:17:10.387434 10705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:10.387759 10718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:10.390200 10718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:10.395200 10718 catalog_manager.cc:1347] Generated new cluster ID: 754cc41bea844f98a3af9325a7021f69
I20260928 21:17:10.395287 10718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:10.402184 10718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:10.403000 10718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:10.420923 10718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0427117679114510b12d203f88ed6256: Generated new TSK 0
I20260928 21:17:10.421649 10718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:10.868039 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:10.428082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.5.18.254:41913)
0928 21:17:10.428186 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:10.428190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:10.428349 (+ 159us) server_negotiation.cc:407] Connection header received
0928 21:17:10.428530 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:10.428533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:10.428578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:10.428666 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:17:10.428884 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.429445 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.430312 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.430540 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.430624 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.430912 (+ 288us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:17:10.867609 (+436697us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:17:10.867912 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40048: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 21:17:10.868124 10702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40048: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 21:17:10.868162 10728 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:10.428009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:41913 (local address 127.0.0.1:40048)
0928 21:17:10.428275 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:10.428286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:10.428406 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:10.428673 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:10.428676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:10.428684 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:17:10.428791 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:10.428798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.429550 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.429554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:10.430204 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:10.430212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:10.430640 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:10.430644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:10.430736 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:10.430758 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:17:10.867827 (+437069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:17:10.868003 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.18.254:41913: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260928 21:17:10.868721 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10632
2026-09-28T21:17:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (690 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:11 dist-test-slave-rc9f krb5kdc[10740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:11 dist-test-slave-rc9f krb5kdc[10740](info): set up 2 sockets
Sep 28 21:17:11 dist-test-slave-rc9f krb5kdc[10740](info): commencing operation
krb5kdc: starting...
W20260928 21:17:12.057502 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:12.163813 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42721
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:12.267196 10753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:12.267500 10753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:12.267539 10753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:12.271584 10753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:12.271661 10753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:12.271680 10753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:12.271695 10753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:12.271709 10753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:12.276893 10753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:12.277971 10753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:12.279007 10753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:12.286875 10759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.286875 10761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.286875 10758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.286981 10753 server_base.cc:1034] running on GCE node
I20260928 21:17:12.287633 10753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:12.288631 10753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:12.289770 10753 hybrid_clock.cc:648] HybridClock initialized: now 1790630232289744 us; error 38 us; skew 500 ppm
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:12.297824 10753 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:12.298688 10753 webserver.cc:460] Webserver started at http://127.5.18.254:41759/ using document root <none> and password file <none>
I20260928 21:17:12.299192 10753 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:12.299250 10753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:12.299455 10753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:12.301407 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "4333eb6d656f4f70b6fd00af88a8366d"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "84546912a6103f34a95f0a44decd31b3"
server_key_iv: "ec7a9d6f8c5425391f26cbe20490e5cb"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.301959 10753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "4333eb6d656f4f70b6fd00af88a8366d"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "84546912a6103f34a95f0a44decd31b3"
server_key_iv: "ec7a9d6f8c5425391f26cbe20490e5cb"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.305392 10753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:12.307664 10768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.308856 10753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:12.308969 10753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "4333eb6d656f4f70b6fd00af88a8366d"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "84546912a6103f34a95f0a44decd31b3"
server_key_iv: "ec7a9d6f8c5425391f26cbe20490e5cb"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.309063 10753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:12.328047 10753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:12.330695 10753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:12.330860 10753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:12.337246 10753 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:42721
I20260928 21:17:12.337265 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:42721 every 8 connection(s)
I20260928 21:17:12.338261 10753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:12.338428 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10753
I20260928 21:17:12.338531 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:12.338802 5195 external_mini_cluster.cc:1428] Setting key ae7e43388c3a151e8375206ef4e71b99
I20260928 21:17:12.341027 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:12.346371 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d: Bootstrap starting.
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:12.348743 10821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:12.349419 10821 log.cc:824] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:12.351406 10821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d: No bootstrap required, opened a new log
I20260928 21:17:12.351965 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:12.340154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.5.18.254:42721)
0928 21:17:12.340534 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:12.340550 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:12.340584 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:12.341227 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:12.341245 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:12.341586 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:12.341900 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:12.342685 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.343683 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.344506 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.344967 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.345116 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:12.347632 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:12.347637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:12.347648 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:12.347675 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:12.349223 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.349653 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.349658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.349662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.349790 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.350081 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.350084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.350085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.350473 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:12.350627 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:12.350898 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 21:17:12.351163 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260928 21:17:12.354781 10821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } }
I20260928 21:17:12.355000 10821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:12.355055 10821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4333eb6d656f4f70b6fd00af88a8366d, State: Initialized, Role: FOLLOWER
I20260928 21:17:12.355432 10821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } }
I20260928 21:17:12.355548 10821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:12.355607 10821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:12.355700 10821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:12.356586 10821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } }
I20260928 21:17:12.356896 10821 leader_election.cc:302] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4333eb6d656f4f70b6fd00af88a8366d; no voters:
I20260928 21:17:12.357158 10821 leader_election.cc:288] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:12.357252 10826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:12.357463 10826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [term 1 LEADER]: Becoming Leader. State: Replica: 4333eb6d656f4f70b6fd00af88a8366d, State: Running, Role: LEADER
I20260928 21:17:12.357786 10826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } }
I20260928 21:17:12.358271 10821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:12.359295 10828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4333eb6d656f4f70b6fd00af88a8366d. Latest consensus state: current_term: 1 leader_uuid: "4333eb6d656f4f70b6fd00af88a8366d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } } }
I20260928 21:17:12.359292 10827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4333eb6d656f4f70b6fd00af88a8366d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4333eb6d656f4f70b6fd00af88a8366d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42721 } } }
I20260928 21:17:12.359460 10828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:12.359534 10827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:12.359896 10836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:12.362664 10836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:12.367647 10836 catalog_manager.cc:1347] Generated new cluster ID: 568be2b1e5d94b2bbad636c9f56e367e
I20260928 21:17:12.367698 10836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:12.395301 10836 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:12.396221 10836 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:12.407379 10836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4333eb6d656f4f70b6fd00af88a8366d: Generated new TSK 0
I20260928 21:17:12.408048 10836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:12.443666 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:12.570982 10847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:12.571241 10847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:12.571287 10847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:12.575277 10847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:12.575330 10847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:12.575399 10847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:12.580590 10847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:12.581588 10847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:12.582847 10847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:12.592370 10852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.592377 10853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.592377 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.592582 10847 server_base.cc:1034] running on GCE node
I20260928 21:17:12.592988 10847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:12.593611 10847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:12.594760 10847 hybrid_clock.cc:648] HybridClock initialized: now 1790630232594737 us; error 34 us; skew 500 ppm
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:12.597213 10847 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:12.598023 10847 webserver.cc:460] Webserver started at http://127.5.18.193:45675/ using document root <none> and password file <none>
I20260928 21:17:12.598524 10847 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:12.598582 10847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:12.598768 10847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:12.600672 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "28b19cf4284c43c7af3c86d696ce82ff"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "0644f8981c60153b9a969a0c3f526f4a"
server_key_iv: "056cdaefa91b5fd1f6d51c0a24a3e2fe"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.601183 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "28b19cf4284c43c7af3c86d696ce82ff"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "0644f8981c60153b9a969a0c3f526f4a"
server_key_iv: "056cdaefa91b5fd1f6d51c0a24a3e2fe"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.604571 10847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:12.606748 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.607862 10847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:12.607962 10847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "28b19cf4284c43c7af3c86d696ce82ff"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "0644f8981c60153b9a969a0c3f526f4a"
server_key_iv: "056cdaefa91b5fd1f6d51c0a24a3e2fe"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.608059 10847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:12.620397 10847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:12.622993 10847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:12.623129 10847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:12.623625 10847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:12.624325 10847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:12.624368 10847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.624402 10847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:12.624423 10847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.632223 10847 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:34705
I20260928 21:17:12.632259 10975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:34705 every 8 connection(s)
I20260928 21:17:12.633055 10847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:12.639346 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10847
I20260928 21:17:12.639477 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:12.639755 5195 external_mini_cluster.cc:1428] Setting key 2c6ed2b2364a3f11b0bcb02615784560
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:12.647162 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:12.634803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:48679 (local address 127.5.18.254:42721)
0928 21:17:12.634969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:12.634973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:12.635773 (+ 800us) server_negotiation.cc:407] Connection header received
0928 21:17:12.637119 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:12.637123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:12.637178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:12.637247 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:12.638422 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.638943 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.640400 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.640733 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.640835 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:12.643515 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:12.643518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:12.643520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:12.643550 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:12.645028 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.645639 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.645643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.645645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.645734 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.646076 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.646078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.646079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.646255 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:12.646328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:12.646937 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 21:17:12.647038 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260928 21:17:12.648361 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:12.635174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42721 (local address 127.5.18.193:48679)
0928 21:17:12.635623 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:12.635667 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:12.636924 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:12.637421 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:12.637428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:12.637804 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:12.638285 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:12.638298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.639055 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.639058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:12.640286 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:12.640296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.640925 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.640930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:12.641111 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:12.641713 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:12.641731 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:12.643341 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:12.645145 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:12.645150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:12.645162 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:12.645489 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:12.645818 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:12.645820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:12.645821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:12.645980 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:12.646350 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:12.646356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:12.646692 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 21:17:12.646922 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 21:17:12.647291 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:17:12.649547 10976 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42721
I20260928 21:17:12.649804 10976 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:12.650286 10976 heartbeater.cc:511] Master 127.5.18.254:42721 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:12.651806 10785 ts_manager.cc:194] Registered new tserver with Master: 28b19cf4284c43c7af3c86d696ce82ff (127.5.18.193:34705)
I20260928 21:17:12.652930 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:48679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:12.666118 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:12.771842 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:12.772099 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:12.772164 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:12.776088 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:12.776146 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:12.776242 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:12.781262 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:12.782230 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:12.783185 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:12.792021 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.792009 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.792197 10981 server_base.cc:1034] running on GCE node
W20260928 21:17:12.791999 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.792660 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:12.793200 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:12.794340 10981 hybrid_clock.cc:648] HybridClock initialized: now 1790630232794325 us; error 33 us; skew 500 ppm
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:12.796787 10981 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:12.797531 10981 webserver.cc:460] Webserver started at http://127.5.18.194:45125/ using document root <none> and password file <none>
I20260928 21:17:12.798029 10981 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:12.798094 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:12.798243 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:12.800048 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "29ee46f03a7b4525a701787adc95e4b1"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "76d6447d5588dd3b59e66690a8803c31"
server_key_iv: "1e85169571f6c37b97abba12449bafb4"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.800511 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "29ee46f03a7b4525a701787adc95e4b1"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "76d6447d5588dd3b59e66690a8803c31"
server_key_iv: "1e85169571f6c37b97abba12449bafb4"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.803870 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 21:17:12.806066 10996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.807180 10981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:12.807263 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "29ee46f03a7b4525a701787adc95e4b1"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "76d6447d5588dd3b59e66690a8803c31"
server_key_iv: "1e85169571f6c37b97abba12449bafb4"
server_key_version: "encryptionkey@0"
I20260928 21:17:12.807355 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:12.824970 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:12.827699 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:12.827862 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:12.828393 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:12.829171 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:12.829216 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.829250 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:12.829270 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:12.836970 10981 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:35541
I20260928 21:17:12.837037 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:35541 every 8 connection(s)
I20260928 21:17:12.837797 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:12.841104 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 10981
I20260928 21:17:12.841195 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:12.841449 5195 external_mini_cluster.cc:1428] Setting key 5cfc6e577fa2f71173cc4cba82aa161b
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:12.851195 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:12.839379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:38823 (local address 127.5.18.254:42721)
0928 21:17:12.839500 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:12.839503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:12.840272 (+ 769us) server_negotiation.cc:407] Connection header received
0928 21:17:12.841690 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:12.841694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:12.841749 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:12.841832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:12.843107 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.843646 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.844961 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.845283 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.845389 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:12.847615 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:12.847618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:12.847621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:12.847649 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:12.849166 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.849641 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.849645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.849647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.849749 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:12.850061 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:12.850065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:12.850067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:12.850313 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:12.850438 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:12.850902 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 21:17:12.851006 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:17:12.851960 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:12.839699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42721 (local address 127.5.18.194:38823)
0928 21:17:12.840125 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:12.840165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:12.841462 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:12.842003 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:12.842013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:12.842375 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:12.842934 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:12.842954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.843770 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.843774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:12.844804 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:12.844813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:12.845409 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:12.845412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:12.845547 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:12.846086 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:12.846101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:12.847456 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:12.849266 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:12.849271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:12.849282 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:12.849508 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:12.849843 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:12.849844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:12.849846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:12.849966 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:12.850429 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:12.850434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:12.850656 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 21:17:12.850804 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 21:17:12.851051 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:12.853168 11110 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42721
I20260928 21:17:12.853412 11110 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:12.853893 11110 heartbeater.cc:511] Master 127.5.18.254:42721 requested a full tablet report, sending...
I20260928 21:17:12.854820 10785 ts_manager.cc:194] Registered new tserver with Master: 29ee46f03a7b4525a701787adc95e4b1 (127.5.18.194:35541)
I20260928 21:17:12.855360 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:38823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:12.866719 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:33693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:12.972590 11115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:12.972826 11115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:12.972880 11115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:12.976781 11115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:12.976831 11115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:12.976908 11115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:12.981940 11115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:33693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:12.982858 11115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:12.983765 11115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:12.992515 11120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:12.992502 11121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.992594 11115 server_base.cc:1034] running on GCE node
W20260928 21:17:12.992502 11123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:12.993076 11115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:12.993692 11115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:12.994849 11115 hybrid_clock.cc:648] HybridClock initialized: now 1790630232994831 us; error 33 us; skew 500 ppm
Sep 28 21:17:12 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:12.997354 11115 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:12.998171 11115 webserver.cc:460] Webserver started at http://127.5.18.195:40561/ using document root <none> and password file <none>
I20260928 21:17:12.998673 11115 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:12.998735 11115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:12.998907 11115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:13.000784 11115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "017be6b81cae41e1894d7c17d0faad64"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "7cadf9f27f61d393a2f8b91857f3cf43"
server_key_iv: "8f3d276bc00ee6e4754ee3aed19bfc5c"
server_key_version: "encryptionkey@0"
I20260928 21:17:13.001295 11115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "017be6b81cae41e1894d7c17d0faad64"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "7cadf9f27f61d393a2f8b91857f3cf43"
server_key_iv: "8f3d276bc00ee6e4754ee3aed19bfc5c"
server_key_version: "encryptionkey@0"
I20260928 21:17:13.004678 11115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:13.006901 11130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:13.007982 11115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 21:17:13.008080 11115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "017be6b81cae41e1894d7c17d0faad64"
format_stamp: "Formatted at 2026-09-28 21:17:12 on dist-test-slave-rc9f"
server_key: "7cadf9f27f61d393a2f8b91857f3cf43"
server_key_iv: "8f3d276bc00ee6e4754ee3aed19bfc5c"
server_key_version: "encryptionkey@0"
I20260928 21:17:13.008167 11115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:13.022303 11115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:13.025084 11115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:13.025241 11115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:13.025782 11115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:13.026517 11115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:13.026571 11115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:13.026620 11115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:13.026646 11115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:13.034562 11115 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:43951
I20260928 21:17:13.034605 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:43951 every 8 connection(s)
I20260928 21:17:13.035346 11115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:13.041797 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11115
I20260928 21:17:13.041903 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:13.042153 5195 external_mini_cluster.cc:1428] Setting key 5687d3d8554bf9b988d293327dd9e569
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630232, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:13.047586 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:13.036966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:51223 (local address 127.5.18.254:42721)
0928 21:17:13.037074 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.037077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.037823 (+ 746us) server_negotiation.cc:407] Connection header received
0928 21:17:13.039065 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.039067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.039113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.039182 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.040193 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.040698 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.041781 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.042059 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.042150 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:13.044632 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:13.044635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:13.044637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:13.044662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:13.046011 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.046451 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.046453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.046454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.046518 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.046755 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.046757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.046758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.046910 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:13.046997 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:13.047385 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 21:17:13.047476 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:13.048319 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:13.037251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42721 (local address 127.5.18.195:51223)
0928 21:17:13.037677 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:13.037716 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:13.038885 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:13.039318 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:13.039325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:13.039637 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:13.040091 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:13.040103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.040779 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.040781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:13.041666 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:13.041674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.042187 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.042191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:13.042347 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:13.042841 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:13.042856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:13.044468 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:13.046123 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:13.046128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:13.046139 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:13.046378 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:13.046578 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:13.046580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:13.046581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:13.046696 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:13.046994 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:13.046998 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:13.047211 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 21:17:13.047355 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 21:17:13.047577 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260928 21:17:13.049286 11244 heartbeater.cc:347] Connected to a master server at 127.5.18.254:42721
I20260928 21:17:13.049504 11244 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:13.049995 11244 heartbeater.cc:511] Master 127.5.18.254:42721 requested a full tablet report, sending...
I20260928 21:17:13.050872 10785 ts_manager.cc:194] Registered new tserver with Master: 017be6b81cae41e1894d7c17d0faad64 (127.5.18.195:43951)
I20260928 21:17:13.051388 10785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:51223
I20260928 21:17:13.055631 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/test-admin.keytab.
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.069526 5195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:17:13.072064 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:13.070777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.5.18.193:34705)
0928 21:17:13.071117 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.071125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.071147 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:17:13.071193 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.071198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.071370 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.071492 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.071892 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32972: BlockingRecv error: recv got EOF from 127.0.0.1:32972 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":154,"threads_started":1}
W20260928 21:17:13.072142 5195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.073083 5195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.073901 5195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:17:13.082607 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:13.074667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.5.18.193:34705)
0928 21:17:13.074778 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.074782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.074867 (+ 85us) server_negotiation.cc:407] Connection header received
0928 21:17:13.074950 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.074952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.074987 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.075045 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.075473 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.076234 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.076965 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.077324 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.077399 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:13.079426 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:13.079434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:13.079447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:13.079469 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:13.080966 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.081351 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.081357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.081358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.081432 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.081698 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.081701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.081703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.081928 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:13.082117 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:13.082323 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:17:13.082464 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 21:17:13.086071 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:13.085162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.5.18.193:34705)
0928 21:17:13.085259 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.085262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.085423 (+ 161us) server_negotiation.cc:407] Connection header received
0928 21:17:13.085510 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.085512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.085554 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.085674 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.085983 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32990: BlockingRecv error: recv got EOF from 127.0.0.1:32990 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 21:17:13.086344 5195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.087282 5195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.088059 5195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:17:13.095252 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:13.088755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.5.18.193:34705)
0928 21:17:13.088871 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.088874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.088931 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:17:13.089022 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.089024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.089056 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.089108 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.089484 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.090025 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.090729 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.090957 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.091032 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:13.092899 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:13.092903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:13.092905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:13.092928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:13.094079 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.094421 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.094424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.094425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.094491 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.094675 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.094677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.094678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.094808 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:13.094890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:13.095035 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:17:13.095114 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:17:13.097496 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:13.096652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.5.18.193:34705)
0928 21:17:13.096739 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.096742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.096864 (+ 122us) server_negotiation.cc:407] Connection header received
0928 21:17:13.096973 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.096975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.097007 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.097060 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.097417 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33006: BlockingRecv error: recv got EOF from 127.0.0.1:33006 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 21:17:13.097625 5195 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790630206151656-5195-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.098450 5195 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:13.099197 5195 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[10740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:17:13.106140 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:13.099898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.5.18.193:34705)
0928 21:17:13.100002 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:13.100006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:13.100078 (+ 72us) server_negotiation.cc:407] Connection header received
0928 21:17:13.100151 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:13.100152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:13.100182 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:13.100238 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:13.100607 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.101063 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.101774 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:13.101978 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:13.102046 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:13.103852 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:13.103855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:13.103857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:13.103877 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:13.104907 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.105230 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.105233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.105234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.105295 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:13.105464 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:13.105466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:13.105467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:13.105641 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:13.105728 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:13.105921 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:17:13.106006 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:17:13.106959 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10847
I20260928 21:17:13.112109 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10981
I20260928 21:17:13.116914 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11115
I20260928 21:17:13.121225 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 10753
2026-09-28T21:17:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2016 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[11264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[11264](info): set up 2 sockets
Sep 28 21:17:13 dist-test-slave-rc9f krb5kdc[11264](info): commencing operation
krb5kdc: starting...
W20260928 21:17:14.069491 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab.
Entry for principal oryx/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab.
I20260928 21:17:14.178395 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37051
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab
--principal=oryx/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:14.285498 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:14.285789 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:14.285845 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:14.289839 11277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:14.289903 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:14.289929 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:14.289948 11277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:14.289968 11277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:14.295154 11277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab
--principal=oryx/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:14.296164 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:14.297129 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:14.305271 11282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:14.305279 11285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.305315 11277 server_base.cc:1034] running on GCE node
W20260928 21:17:14.305281 11283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.306095 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:14.307185 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:14.308336 11277 hybrid_clock.cc:648] HybridClock initialized: now 1790630234308311 us; error 41 us; skew 500 ppm
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:14.310967 11277 init.cc:377] Logged in from keytab as oryx/127.5.18.254@KRBTEST.COM (short username oryx)
I20260928 21:17:14.311839 11277 webserver.cc:460] Webserver started at http://127.5.18.254:35839/ using document root <none> and password file <none>
I20260928 21:17:14.312376 11277 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:14.312439 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:14.312670 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:14.314715 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "e3a0ad3c30d647e68e493adb3541ef0a"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "f37ada3420f53ac7665cd17b5deee894"
server_key_iv: "430effd6398a6d8174b251dcac0a64f0"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.315244 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "e3a0ad3c30d647e68e493adb3541ef0a"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "f37ada3420f53ac7665cd17b5deee894"
server_key_iv: "430effd6398a6d8174b251dcac0a64f0"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.318724 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:14.320938 11292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.322165 11277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:14.322278 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "e3a0ad3c30d647e68e493adb3541ef0a"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "f37ada3420f53ac7665cd17b5deee894"
server_key_iv: "430effd6398a6d8174b251dcac0a64f0"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.322365 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:14.348811 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:14.352038 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:14.352213 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:14.358610 11277 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:37051
I20260928 21:17:14.358878 11344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:37051 every 8 connection(s)
I20260928 21:17:14.359527 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:14.362228 11345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:14.363413 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11277
I20260928 21:17:14.363514 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:14.363771 5195 external_mini_cluster.cc:1428] Setting key d950f01e0adf10ed4c76fb5177c4c2be
I20260928 21:17:14.368444 11345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Bootstrap starting.
I20260928 21:17:14.370918 11345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:14.371694 11345 log.cc:824] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:14.373703 11345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: No bootstrap required, opened a new log
I20260928 21:17:14.376953 11345 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } }
I20260928 21:17:14.377166 11345 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:14.377207 11345 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a0ad3c30d647e68e493adb3541ef0a, State: Initialized, Role: FOLLOWER
I20260928 21:17:14.377580 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:14.365052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.5.18.254:37051)
0928 21:17:14.365435 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:14.365454 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:14.365495 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:17:14.366362 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:14.366387 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:14.366794 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:14.367170 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:14.367796 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.369195 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.370041 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.370684 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.370831 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:14.373262 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:14.373268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:14.373283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:14.373312 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:14.374942 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.375337 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.375342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.375346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.375452 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.375695 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.375698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.375700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.376066 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:14.376248 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:14.376513 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:17:14.376784 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260928 21:17:14.377703 11345 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } }
I20260928 21:17:14.377856 11345 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:14.377928 11345 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:14.378019 11345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:14.378944 11345 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } }
I20260928 21:17:14.379282 11345 leader_election.cc:302] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a0ad3c30d647e68e493adb3541ef0a; no voters:
I20260928 21:17:14.379596 11345 leader_election.cc:288] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:14.379684 11350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:14.379910 11350 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 LEADER]: Becoming Leader. State: Replica: e3a0ad3c30d647e68e493adb3541ef0a, State: Running, Role: LEADER
I20260928 21:17:14.380234 11350 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } }
I20260928 21:17:14.380723 11345 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:14.382176 11352 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a0ad3c30d647e68e493adb3541ef0a. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } }
I20260928 21:17:14.382432 11351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } }
I20260928 21:17:14.382530 11352 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:14.382633 11351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
W20260928 21:17:14.385157 11366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:17:14.385221 11366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:17:14.385301 11365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:14.386010 11365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:14.390901 11365 catalog_manager.cc:1347] Generated new cluster ID: 4b47dabd25294a92a45f07ccbfcdd448
I20260928 21:17:14.390978 11365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:14.402601 11365 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:14.403430 11365 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:14.408882 11365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Generated new TSK 0
I20260928 21:17:14.409494 11365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
I20260928 21:17:14.438472 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:14.543455 11371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:14.543706 11371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:14.543747 11371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:14.547646 11371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:14.547704 11371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:14.547760 11371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:14.552826 11371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:14.553859 11371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:14.554838 11371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:14.563800 11377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.563934 11371 server_base.cc:1034] running on GCE node
W20260928 21:17:14.563824 11376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:14.563886 11379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.564426 11371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:14.565037 11371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:14.566195 11371 hybrid_clock.cc:648] HybridClock initialized: now 1790630234566180 us; error 30 us; skew 500 ppm
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:14.568871 11371 init.cc:377] Logged in from keytab as oryx/127.5.18.193@KRBTEST.COM (short username oryx)
I20260928 21:17:14.569761 11371 webserver.cc:460] Webserver started at http://127.5.18.193:44009/ using document root <none> and password file <none>
I20260928 21:17:14.570297 11371 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:14.570369 11371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:14.570580 11371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:14.572597 11371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "c60bbcd34a454f692bf138e5c10d48b2"
server_key_iv: "f5116a6e52368819ab8ad531f193c860"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.573151 11371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "c60bbcd34a454f692bf138e5c10d48b2"
server_key_iv: "f5116a6e52368819ab8ad531f193c860"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.576897 11371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:14.579288 11386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.580485 11371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:17:14.580595 11371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "c60bbcd34a454f692bf138e5c10d48b2"
server_key_iv: "f5116a6e52368819ab8ad531f193c860"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.580693 11371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:14.597291 11371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:14.600054 11371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:14.600219 11371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:14.600759 11371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:14.601552 11371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:14.601637 11371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.601691 11371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:14.601713 11371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.610636 11371 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:46693
I20260928 21:17:14.610659 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:46693 every 8 connection(s)
I20260928 21:17:14.611552 11371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:14.613812 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11371
I20260928 21:17:14.613950 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:14.614295 5195 external_mini_cluster.cc:1428] Setting key ec2196f9606f654301db12cfeb276298
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:17:14.625474 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:14.613675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56947 (local address 127.5.18.254:37051)
0928 21:17:14.613792 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:14.613795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:14.615061 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 21:17:14.616051 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:14.616056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:14.616114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:14.616214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:14.617687 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.618220 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.619359 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.619642 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.619731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:14.622136 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:14.622140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:14.622143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:14.622171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:14.623480 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.623953 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.623957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.623959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.624041 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.624349 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.624352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.624354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.624529 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:14.624612 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:14.625185 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 21:17:14.625309 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:14.626432 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:14.614181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:56947)
0928 21:17:14.614928 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:14.614971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:14.615817 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:14.616397 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:14.616406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:14.616866 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:14.617528 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:14.617546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.618335 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.618338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:14.619245 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:14.619254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.619777 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.619781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:14.619950 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:14.620483 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:14.620499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:14.621977 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:14.623591 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:14.623597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:14.623612 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:14.623853 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:14.624119 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:14.624122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:14.624124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:14.624257 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:14.624625 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:14.624632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:14.624918 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 21:17:14.625129 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 21:17:14.625451 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":168,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:14.627791 11500 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:14.628017 11500 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:14.628501 11500 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:14.630000 11309 ts_manager.cc:194] Registered new tserver with Master: 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:14.631099 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:56947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
I20260928 21:17:14.641772 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:14.746968 11505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:14.747227 11505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:14.747268 11505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:14.751160 11505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:14.751214 11505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:14.751271 11505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:14.756263 11505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:14.757196 11505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:14.758184 11505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:14.767066 11511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:14.767062 11510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.767213 11505 server_base.cc:1034] running on GCE node
W20260928 21:17:14.767062 11513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.767685 11505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:14.768280 11505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:14.769444 11505 hybrid_clock.cc:648] HybridClock initialized: now 1790630234769423 us; error 31 us; skew 500 ppm
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:14.772066 11505 init.cc:377] Logged in from keytab as oryx/127.5.18.194@KRBTEST.COM (short username oryx)
I20260928 21:17:14.772830 11505 webserver.cc:460] Webserver started at http://127.5.18.194:42219/ using document root <none> and password file <none>
I20260928 21:17:14.773326 11505 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:14.773379 11505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:14.773545 11505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:14.775501 11505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "ffb87801d7964812bf3162738fcccb5e"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "44f45e8e12d3ad262e12de509e1bd9bb"
server_key_iv: "8920b4af3f258bb35b69d6e55b220728"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.775996 11505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "ffb87801d7964812bf3162738fcccb5e"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "44f45e8e12d3ad262e12de509e1bd9bb"
server_key_iv: "8920b4af3f258bb35b69d6e55b220728"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.779547 11505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:14.781929 11520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.783149 11505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:14.783244 11505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "ffb87801d7964812bf3162738fcccb5e"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "44f45e8e12d3ad262e12de509e1bd9bb"
server_key_iv: "8920b4af3f258bb35b69d6e55b220728"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.783350 11505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:14.797729 11505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:14.800580 11505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:14.800731 11505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:14.801216 11505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:14.801959 11505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:14.802001 11505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.802047 11505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:14.802066 11505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.809845 11505 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43835
I20260928 21:17:14.809878 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43835 every 8 connection(s)
I20260928 21:17:14.810642 11505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:14.817162 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11505
I20260928 21:17:14.817273 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:14.817507 5195 external_mini_cluster.cc:1428] Setting key 6ede74a438f9870c0438f47ab431f391
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:14.822851 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:14.812232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:36039 (local address 127.5.18.254:37051)
0928 21:17:14.812357 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:14.812362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:14.813345 (+ 983us) server_negotiation.cc:407] Connection header received
0928 21:17:14.814168 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:14.814171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:14.814216 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:14.814285 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:14.815312 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.815816 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.816825 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.817034 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.817116 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:14.819505 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:14.819508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:14.819511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:14.819542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:14.820955 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.821415 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.821419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.821421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.821506 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:14.821803 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:14.821806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:14.821807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:14.822004 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:14.822097 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:14.822563 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 21:17:14.822685 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:14.823493 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:14.812617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:36039)
0928 21:17:14.813211 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:14.813249 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:14.813999 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:14.814426 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:14.814432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:14.814767 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:14.815206 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:14.815219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.815901 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.815903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:14.816741 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:14.816749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:14.817173 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:14.817177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:14.817349 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:14.817910 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:14.817928 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:14.819357 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:14.821060 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:14.821064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:14.821075 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:14.821316 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:14.821601 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:14.821603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:14.821605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:14.821728 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:14.822105 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:14.822109 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:14.822308 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 21:17:14.822471 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 21:17:14.822714 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":101,"threads_started":1}
I20260928 21:17:14.824469 11634 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:14.824681 11634 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:14.825105 11634 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:14.826042 11309 ts_manager.cc:194] Registered new tserver with Master: ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835)
WARNING: no policy specified for oryx/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:14.826566 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:36039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
I20260928 21:17:14.841166 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:14.948345 11639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:14.948699 11639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:14.948771 11639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:14.953100 11639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:14.953158 11639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:14.953217 11639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:14.958302 11639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:14.959256 11639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:14.960202 11639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:17:14.969408 11639 server_base.cc:1034] running on GCE node
W20260928 21:17:14.969347 11644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:14.969340 11645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:14.969352 11647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:14.969980 11639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:14.970539 11639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:14.971693 11639 hybrid_clock.cc:648] HybridClock initialized: now 1790630234971677 us; error 30 us; skew 500 ppm
Sep 28 21:17:14 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:14.974221 11639 init.cc:377] Logged in from keytab as oryx/127.5.18.195@KRBTEST.COM (short username oryx)
I20260928 21:17:14.975175 11639 webserver.cc:460] Webserver started at http://127.5.18.195:39803/ using document root <none> and password file <none>
I20260928 21:17:14.975813 11639 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:14.975924 11639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:14.976127 11639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:14.978063 11639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "57163cb0239e46be85b86e8c04bb7dc9"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "cf469e979627da0a1baec84c41df9278"
server_key_iv: "31493d1f4553cf7c421e9c52109e9e87"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.978564 11639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "57163cb0239e46be85b86e8c04bb7dc9"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "cf469e979627da0a1baec84c41df9278"
server_key_iv: "31493d1f4553cf7c421e9c52109e9e87"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.982029 11639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:17:14.984254 11654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:14.985421 11639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:14.985517 11639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "57163cb0239e46be85b86e8c04bb7dc9"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "cf469e979627da0a1baec84c41df9278"
server_key_iv: "31493d1f4553cf7c421e9c52109e9e87"
server_key_version: "encryptionkey@0"
I20260928 21:17:14.985628 11639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:14.997673 11639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.000044 11639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:15.000198 11639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.000692 11639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:15.001431 11639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:15.001482 11639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.001531 11639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:15.001555 11639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.009486 11639 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:36429
I20260928 21:17:15.009507 11767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:36429 every 8 connection(s)
I20260928 21:17:15.010350 11639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:15.016176 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11639
I20260928 21:17:15.016273 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:15.016518 5195 external_mini_cluster.cc:1428] Setting key e56cb4bdbc0df0203184e2666bf5b852
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:15.022399 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.011999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:58997 (local address 127.5.18.254:37051)
0928 21:17:15.012126 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.012129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.013107 (+ 978us) server_negotiation.cc:407] Connection header received
0928 21:17:15.013969 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.013971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.014026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.014102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.015212 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.015742 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.016917 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.017181 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.017281 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.019650 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.019653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.019655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.019679 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.020773 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.021203 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.021205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.021206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.021272 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.021506 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.021508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.021509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.021716 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.021786 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.022194 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.022287 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:17:15.023139 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.012350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:58997)
0928 21:17:15.012953 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.012992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.013747 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.014251 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.014257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.014593 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.015067 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.015080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.015832 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.015834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.016780 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.016792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.017329 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.017333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.017500 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.018084 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.018099 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.019508 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.020890 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.020894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.020905 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.021133 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.021330 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.021332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.021333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.021448 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.021793 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.021797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.022021 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.022165 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.022391 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":102,"threads_started":1}
I20260928 21:17:15.024109 11768 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:15.024315 11768 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.024744 11768 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:15.025683 11309 ts_manager.cc:194] Registered new tserver with Master: 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429)
I20260928 21:17:15.026233 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:58997
I20260928 21:17:15.029934 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.18.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.253.keytab.
Entry for principal oryx/127.5.18.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.253.keytab.
I20260928 21:17:15.052371 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:44807
--webserver_interface=127.5.18.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.253.keytab
--principal=oryx/127.5.18.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:37051,127.5.18.253:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:15.156569 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:15.156822 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:15.156872 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:15.160734 11773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:15.160785 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:15.160810 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:15.160830 11773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:15.160851 11773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:15.165879 11773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:37051,127.5.18.253:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.253.keytab
--principal=oryx/127.5.18.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.18.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
W20260928 21:17:15.166065 11773 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.18.254:37051,127.5.18.253:44807'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:17:15.166842 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:15.167759 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.175500 11779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.175475 11778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.175607 11773 server_base.cc:1034] running on GCE node
W20260928 21:17:15.175510 11781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.176097 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.177117 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.178261 11773 hybrid_clock.cc:648] HybridClock initialized: now 1790630235178249 us; error 33 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.180732 11773 init.cc:377] Logged in from keytab as oryx/127.5.18.253@KRBTEST.COM (short username oryx)
I20260928 21:17:15.181640 11773 webserver.cc:460] Webserver started at http://127.5.18.253:45465/ using document root <none> and password file <none>
I20260928 21:17:15.182160 11773 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.182248 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.182456 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:15.184512 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/instance:
uuid: "dc5b06321624466d94741b96c4dac7df"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "4093d4409247a0a0c8d3b32e8b56011f"
server_key_iv: "5622583097a7229f3fa2d1aab4f9b5e4"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.185038 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal/instance:
uuid: "dc5b06321624466d94741b96c4dac7df"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "4093d4409247a0a0c8d3b32e8b56011f"
server_key_iv: "5622583097a7229f3fa2d1aab4f9b5e4"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.188585 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:15.190944 11788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.192101 11773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:15.192198 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "dc5b06321624466d94741b96c4dac7df"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "4093d4409247a0a0c8d3b32e8b56011f"
server_key_iv: "5622583097a7229f3fa2d1aab4f9b5e4"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.192293 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.205014 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.207736 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:15.207896 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.214205 11773 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.253:44807
I20260928 21:17:15.214499 11840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.253:44807 every 8 connection(s)
I20260928 21:17:15.215122 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
I20260928 21:17:15.217473 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11773
I20260928 21:17:15.217846 11841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:15.222070 11841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.253@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:15.231753 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.218894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.5.18.253:44807)
0928 21:17:15.219307 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.219322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.219355 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:17:15.219979 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.219995 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.220315 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.220610 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.221378 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.222382 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.223531 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.224202 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.224728 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.226991 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.226995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.227004 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.227037 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.228697 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.229193 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.229199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.229205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.229324 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.229630 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.229633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.229635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.229986 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.230302 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.230563 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.230831 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":144,"threads_started":1}
I20260928 21:17:15.232795 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.223827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56650)
0928 21:17:15.224167 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.224204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.224309 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.224565 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.224570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.224932 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.225117 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.225124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.225950 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.225954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.226657 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.226665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.227103 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.227106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.227204 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.227696 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.227709 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.229225 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.231120 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.231127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.231132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.231386 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.231656 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.231659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.231661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.231750 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.232272 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.232277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.232370 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.232492 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.232623 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260928 21:17:15.232849 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.223889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.5.18.254:37051)
0928 21:17:15.224001 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.224006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.224276 (+ 270us) server_negotiation.cc:407] Connection header received
0928 21:17:15.224399 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.224402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.224446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.224510 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.225225 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.225812 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.226760 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.227005 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.227093 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.229372 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.229377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.229379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.229410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.231022 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.231476 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.231479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.231480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.231560 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.231841 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.231845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.231847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.232145 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.232255 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.232569 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.232689 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:15.234082 11841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.253@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
I20260928 21:17:15.241230 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.234498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40466)
0928 21:17:15.234581 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.234593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.234655 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.234868 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.234870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.234995 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.235089 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.235094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.236135 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.236138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.236672 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.236681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.237083 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.237086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.237158 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.237552 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.237590 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.238678 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.240116 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.240118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.240120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.240291 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.240502 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.240504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.240506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.240571 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.240836 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.240839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.240915 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.240998 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.241099 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:17:15.241302 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.234598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.5.18.253:44807)
0928 21:17:15.234696 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.234699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.234709 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:17:15.234734 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.234736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.234766 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.234822 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.235186 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.235678 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.236776 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.236990 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.237064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.238801 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.238804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.238806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.238825 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.240001 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.240374 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.240377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.240378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.240440 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.240632 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.240634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.240635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.240772 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.240837 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.241047 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.241147 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":38}
I20260928 21:17:15.243839 11841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Bootstrap starting.
I20260928 21:17:15.246124 11841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:15.246829 11841 log.cc:824] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:15.248566 11841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: No bootstrap required, opened a new log
I20260928 21:17:15.251542 11841 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } }
I20260928 21:17:15.251760 11841 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:15.251821 11841 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5b06321624466d94741b96c4dac7df, State: Initialized, Role: FOLLOWER
I20260928 21:17:15.252250 11841 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } }
I20260928 21:17:15.252676 11847 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } } }
I20260928 21:17:15.252781 11847 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:15.253376 11841 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:17:15.257652 11858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:17:15.257715 11858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:17:15.260039 11773 master_runner.cc:428] Detected that this master dc5b06321624466d94741b96c4dac7df is joining an existing cluster
I20260928 21:17:15.260087 11773 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.18.253:44807
I20260928 21:17:15.268993 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.261842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56658)
0928 21:17:15.262087 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.262100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.262192 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.262450 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.262453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.262645 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.262751 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.262757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.263766 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.263768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.264339 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.264345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.264902 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.264904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.264963 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.265424 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.265436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.265778 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.267428 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.267432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.267434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.267691 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.267991 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.267995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.267996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.268093 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.268484 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.268487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.268595 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.268694 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.268812 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260928 21:17:15.269024 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.261932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.5.18.254:37051)
0928 21:17:15.262040 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.262043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.262172 (+ 129us) server_negotiation.cc:407] Connection header received
0928 21:17:15.262281 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.262284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.262337 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.262420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.262874 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.263676 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.264442 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.264787 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.264908 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.265907 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.265911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.265914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.265941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.267329 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.267793 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.267797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.267799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.267899 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.268181 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.268184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.268186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.268393 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.268489 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.268751 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.268873 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:17:15.273842 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.262376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40482)
0928 21:17:15.266517 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.266539 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.266622 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.267449 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.267452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.268009 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.268121 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.268127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.268901 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.268904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.269616 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.269627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.270007 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.270011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.270114 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.270504 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.270518 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.270786 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.272538 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.272541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.272543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.272710 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.272929 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.272932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.272934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.273001 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.273313 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.273316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.273393 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.273506 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.273651 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4085,"thread_start_us":74,"threads_started":1}
I20260928 21:17:15.273896 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.262245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.5.18.253:44807)
0928 21:17:15.262331 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.262334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.266610 (+ 4276us) server_negotiation.cc:407] Connection header received
0928 21:17:15.266750 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.266753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.266798 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.266885 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.268224 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.268795 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.269716 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.269923 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.270003 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.270897 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.270900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.270903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.270924 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.272450 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.272786 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.272789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.272791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.272858 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.273080 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.273083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.273084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.273234 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.273313 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.273615 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.273727 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.254@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
I20260928 21:17:15.283799 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.274451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.5.18.253:44807)
0928 21:17:15.274556 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.274568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.274580 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:15.274635 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.274638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.274683 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.274752 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.276567 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.277017 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.278227 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.278418 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.278484 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.280761 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.280765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.280766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.280787 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.282181 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.282685 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.282687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.282688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.282758 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.283002 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.283004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.283005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.283190 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.283261 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.283598 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.283685 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:17:15.283927 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.273717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40486)
0928 21:17:15.274251 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.274304 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.274420 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.274892 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.274898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.276316 (+ 1418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.276456 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.276464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.277108 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.277111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.278127 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.278137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.278504 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.278508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.278614 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.279081 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.279095 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.280640 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.282272 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.282281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.282287 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.282604 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.282831 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.282834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.282836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.282932 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.283275 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.283282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.283409 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.283556 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.283743 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":89,"threads_started":1}
I20260928 21:17:15.285404 11309 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.18.253:44807: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dc5b06321624466d94741b96c4dac7df"
member_type: NON_VOTER
last_known_addr {
host: "127.5.18.253"
port: 44807
}
attrs {
promote: true
}
}
dest_uuid: "e3a0ad3c30d647e68e493adb3541ef0a"
cas_config_opid_index: -1
I20260928 21:17:15.286190 11309 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } }
I20260928 21:17:15.288674 11815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:15.289614 11869 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dc5b06321624466d94741b96c4dac7df (127.5.18.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } }
I20260928 21:17:15.290100 11815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Refusing update from remote peer e3a0ad3c30d647e68e493adb3541ef0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:17:15.291501 11869 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.18.253:44807
I20260928 21:17:15.291637 11868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:15.292251 11773 master.cc:537] Master@127.5.18.253:44807 shutting down...
WARNING: no policy specified for oryx/127.5.18.252@KRBTEST.COM; defaulting to no policy
I20260928 21:17:15.293367 11869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } }
I20260928 21:17:15.293466 11869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
W20260928 21:17:15.293673 11294 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 21:17:15.294806 11294 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a -> Peer dc5b06321624466d94741b96c4dac7df (127.5.18.253:44807): Couldn't send request to peer dc5b06321624466d94741b96c4dac7df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.252@KRBTEST.COM" created.
I20260928 21:17:15.296519 11773 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:15.296792 11773 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:15.296892 11773 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: stopping tablet replica
I20260928 21:17:15.302172 11773 master.cc:552] Master@127.5.18.253:44807 shutdown complete.
I20260928 21:17:15.302220 11773 master_runner.cc:305] Clearing existing system tablet
I20260928 21:17:15.302449 11773 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.302500 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.303697 11773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 21:17:15.305081 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.305248 11773 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.305366 11773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 21:17:15.305397 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "dc5b06321624466d94741b96c4dac7df"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "4093d4409247a0a0c8d3b32e8b56011f"
server_key_iv: "5622583097a7229f3fa2d1aab4f9b5e4"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.307605 11773 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.252.keytab.
Entry for principal oryx/127.5.18.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.252.keytab.
I20260928 21:17:15.310002 11773 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:17:15.310097 11773 log.cc:1196] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:17:15.310302 11773 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Deleting consensus metadata
I20260928 21:17:15.310456 11773 master_runner.cc:315] Copying system tablet from 127.5.18.254:37051
I20260928 21:17:15.310572 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:44759
--webserver_interface=127.5.18.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.252.keytab
--principal=oryx/127.5.18.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:15.311512 11773 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: tablet copy: Beginning tablet copy session from remote peer at address 127.5.18.254:37051
I20260928 21:17:15.319476 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.312158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56672)
0928 21:17:15.312505 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.312520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.312646 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.312901 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.312904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.313093 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.313186 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.313191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.314172 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.314174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.314739 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.314747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.315265 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.315268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.315323 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.315819 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.315830 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.316129 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.317936 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.317939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.317941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.318189 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.318482 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.318485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.318487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.318582 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.318982 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.318985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.319083 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.319178 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.319300 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260928 21:17:15.319530 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.312242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.5.18.254:37051)
0928 21:17:15.312365 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.312370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.312610 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:17:15.312734 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.312737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.312790 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.312874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.313275 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.314078 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.314834 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.315169 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.315276 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.316263 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.316267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.316270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.316300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.317832 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.318295 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.318299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.318301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.318399 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.318665 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.318669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.318671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.318892 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.318992 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.319248 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.319373 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:15.320487 11329 tablet_copy_service.cc:140] P e3a0ad3c30d647e68e493adb3541ef0a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dc5b06321624466d94741b96c4dac7df ({username='oryx', principal='oryx/127.5.18.253@KRBTEST.COM'} at 127.0.0.1:56672)
I20260928 21:17:15.320672 11329 tablet_copy_service.cc:161] P e3a0ad3c30d647e68e493adb3541ef0a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dc5b06321624466d94741b96c4dac7df at {username='oryx', principal='oryx/127.5.18.253@KRBTEST.COM'} at 127.0.0.1:56672: session id = dc5b06321624466d94741b96c4dac7df-00000000000000000000000000000000
I20260928 21:17:15.322468 11329 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:17:15.324060 11773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:17:15.326372 11773 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: tablet copy: Starting download of 0 data blocks...
I20260928 21:17:15.326563 11773 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: tablet copy: Starting download of 1 WAL segments...
I20260928 21:17:15.328223 11773 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:17:15.329924 11329 tablet_copy_service.cc:342] P e3a0ad3c30d647e68e493adb3541ef0a: Request end of tablet copy session dc5b06321624466d94741b96c4dac7df-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.18.253@KRBTEST.COM'} at 127.0.0.1:56672
I20260928 21:17:15.330032 11329 tablet_copy_service.cc:433] P e3a0ad3c30d647e68e493adb3541ef0a: ending tablet copy session dc5b06321624466d94741b96c4dac7df-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dc5b06321624466d94741b96c4dac7df
I20260928 21:17:15.337481 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.339128 11884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.339223 11885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.339675 11887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.339826 11773 server_base.cc:1034] running on GCE node
I20260928 21:17:15.339980 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.340276 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.341450 11773 hybrid_clock.cc:648] HybridClock initialized: now 1790630235341436 us; error 38 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.343011 11773 init.cc:377] Logged in from keytab as oryx/127.5.18.253@KRBTEST.COM (short username oryx)
I20260928 21:17:15.343477 11773 webserver.cc:460] Webserver started at http://127.5.18.253:44347/ using document root <none> and password file <none>
I20260928 21:17:15.343685 11773 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.343757 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.344972 11773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:15.345957 11894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.346177 11773 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:15.346278 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "dc5b06321624466d94741b96c4dac7df"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "4093d4409247a0a0c8d3b32e8b56011f"
server_key_iv: "5622583097a7229f3fa2d1aab4f9b5e4"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.346416 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.353421 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.355304 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.359797 11773 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.253:44807
I20260928 21:17:15.360355 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
I20260928 21:17:15.362807 11947 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:17:15.363876 11947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Bootstrap starting.
I20260928 21:17:15.369692 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.253:44807 every 8 connection(s)
I20260928 21:17:15.378505 11947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:17:15.378804 11947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: Bootstrap complete.
I20260928 21:17:15.379314 11947 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } }
I20260928 21:17:15.379444 11947 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc5b06321624466d94741b96c4dac7df, State: Initialized, Role: LEARNER
I20260928 21:17:15.379556 11947 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } }
I20260928 21:17:15.379874 11950 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } }
I20260928 21:17:15.379998 11947 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:15.380069 11950 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: LEARNER
I20260928 21:17:15.382776 11961 catalog_manager.cc:1259] Loaded cluster ID: 4b47dabd25294a92a45f07ccbfcdd448
I20260928 21:17:15.382877 11961 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: loading cluster ID for follower catalog manager: success
I20260928 21:17:15.385040 11961 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: acquiring CA information for follower catalog manager: success
I20260928 21:17:15.386358 11961 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:17:15.429880 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:15.430138 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:15.430179 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:15.434182 11882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:15.434244 11882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:15.434265 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:15.434283 11882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:15.434307 11882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:15.439383 11882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.252.keytab
--principal=oryx/127.5.18.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:44759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.18.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:15.440343 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:15.441318 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.449348 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.449421 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.449400 11882 server_base.cc:1034] running on GCE node
W20260928 21:17:15.449348 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.450129 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.451152 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.452310 11882 hybrid_clock.cc:648] HybridClock initialized: now 1790630235452283 us; error 42 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.454859 11882 init.cc:377] Logged in from keytab as oryx/127.5.18.252@KRBTEST.COM (short username oryx)
I20260928 21:17:15.455694 11882 webserver.cc:460] Webserver started at http://127.5.18.252:34793/ using document root <none> and password file <none>
I20260928 21:17:15.456197 11882 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.456259 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.456451 11882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:15.458448 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/instance:
uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "8ff6817d00e12c1a80e76ceee7eeb35f"
server_key_iv: "26ed72ea0bf8542a874f7791ccb3a46b"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.458959 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal/instance:
uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "8ff6817d00e12c1a80e76ceee7eeb35f"
server_key_iv: "26ed72ea0bf8542a874f7791ccb3a46b"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.462334 11882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:15.464447 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.465615 11882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:15.465724 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "8ff6817d00e12c1a80e76ceee7eeb35f"
server_key_iv: "26ed72ea0bf8542a874f7791ccb3a46b"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.465806 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.480096 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.483132 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:15.483289 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.489557 11882 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.252:44759
I20260928 21:17:15.489872 12028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.252:44759 every 8 connection(s)
I20260928 21:17:15.490506 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
I20260928 21:17:15.493008 12029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:15.496484 12029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:17:15.497164 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 11882
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.252@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
I20260928 21:17:15.508893 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.498327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.5.18.254:37051)
0928 21:17:15.498461 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.498465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.498945 (+ 480us) server_negotiation.cc:407] Connection header received
0928 21:17:15.499617 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.499620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.499664 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.499750 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.500693 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.501504 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.502854 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.503196 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.503297 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.505743 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.505747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.505750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.505779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.506978 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.507449 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.507453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.507454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.507549 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.507816 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.507820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.507821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.508007 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.508082 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.508622 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.508735 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:17:15.509821 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.498374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.5.18.252:44759)
0928 21:17:15.498736 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.498754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.498786 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:17:15.499422 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.499439 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.499652 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.499810 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.500604 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.501799 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.502849 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.503377 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.503461 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.506231 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.506236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.506245 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.506264 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.507798 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.508162 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.508167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.508168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.508235 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.508484 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.508486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.508487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.508724 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.508912 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.509379 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.509496 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260928 21:17:15.509821 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.498374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56682)
0928 21:17:15.498810 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.498853 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.499433 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.499796 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.499801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.500211 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.500552 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.500570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.501647 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.501649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.502735 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.502746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.503316 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.503318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.503457 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.504117 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.504135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.505614 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.507076 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.507084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.507092 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.507358 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.507632 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.507634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.507635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.507744 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.508108 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.508114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.508376 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.508556 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.508772 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"spinlock_wait_cycles":26368,"thread_start_us":130,"threads_started":1}
I20260928 21:17:15.510938 12029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.252@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
I20260928 21:17:15.518014 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.511356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40488)
0928 21:17:15.511455 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.511466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.511522 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.512039 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.512041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.512154 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.512232 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.512236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.513055 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.513057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.513556 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.513580 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.513971 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.513973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.514051 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.514414 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.514423 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.515472 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.516954 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.516956 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.516958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.517115 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.517297 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.517299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.517300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.517358 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.517667 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.517669 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.517742 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.517817 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.517906 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:17:15.518083 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.511622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.5.18.253:44807)
0928 21:17:15.511839 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.511850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.511869 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:17:15.511902 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.511904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.511939 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.512005 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.512312 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.512980 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.513655 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.513896 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.513969 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.515593 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.515596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.515598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.515621 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.516880 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.517176 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.517178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.517179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.517242 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.517408 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.517410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.517411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.517599 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.517682 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.517852 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.517967 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260928 21:17:15.518688 12029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.252@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
I20260928 21:17:15.525164 12032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.519075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37732)
0928 21:17:15.519174 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.519187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.519254 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.519425 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.519427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.519566 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.519625 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.519629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.520273 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.520276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.520751 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.520757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.521141 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.521145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.521258 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.521601 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.521611 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.522715 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.524015 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.524018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.524019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.524182 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.524407 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.524409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.524410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.524469 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.524782 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.524784 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.524850 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.524932 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.525022 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":19}
I20260928 21:17:15.525187 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.519134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.5.18.252:44759)
0928 21:17:15.519215 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.519219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.519230 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:15.519316 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.519319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.519351 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.519406 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.519714 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.520188 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.520841 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.521059 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.521136 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.522822 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.522825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.522828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.522847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.523923 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.524260 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.524263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.524266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.524338 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.524531 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.524534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.524535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.524705 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.524775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.524972 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.525075 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":33}
I20260928 21:17:15.528087 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Bootstrap starting.
I20260928 21:17:15.530289 12029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:15.530957 12029 log.cc:824] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:15.532618 12029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: No bootstrap required, opened a new log
I20260928 21:17:15.535445 12029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } }
I20260928 21:17:15.535636 12029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:15.535681 12029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693f0f79de6a4c5393aa1fd0f10f392e, State: Initialized, Role: FOLLOWER
I20260928 21:17:15.536067 12029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } }
I20260928 21:17:15.536460 12035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } } }
I20260928 21:17:15.536567 12035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:15.537122 12029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:17:15.541370 12046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:17:15.541441 12046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:17:15.544152 11882 master_runner.cc:428] Detected that this master 693f0f79de6a4c5393aa1fd0f10f392e is joining an existing cluster
I20260928 21:17:15.544202 11882 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.18.252:44759
I20260928 21:17:15.553220 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.546047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.5.18.254:37051)
0928 21:17:15.546242 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.546247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.546320 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:17:15.546494 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.546497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.546545 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.546597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.547268 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.547936 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.548872 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.549215 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.549283 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.550481 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.550485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.550487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.550512 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.551781 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.552154 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.552157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.552159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.552236 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.552464 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.552468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.552470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.552646 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.552732 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.552981 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.553086 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 21:17:15.553236 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.545957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56698)
0928 21:17:15.546251 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.546265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.546387 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.546737 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.546740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.546964 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.547108 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.547114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.548046 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.548048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.548755 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.548763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.549437 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.549439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.549500 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.550049 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.550058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.550354 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.551875 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.551878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.551881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.552070 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.552312 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.552315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.552317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.552391 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.552732 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.552735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.552834 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.552917 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.553049 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260928 21:17:15.560273 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.546531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40504)
0928 21:17:15.553369 (+ 6838us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.553382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.553501 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.553852 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.553855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.554009 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.554111 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.554118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.554817 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.554820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.555570 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.555580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.556291 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.556295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.556382 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.556819 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.556832 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.557189 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.558930 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.558932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.558933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.559107 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.559362 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.559365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.559367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.559436 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.559800 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.559802 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.559884 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.559983 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.560110 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6787,"thread_start_us":134,"threads_started":1}
I20260928 21:17:15.560562 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.546398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.5.18.253:44807)
0928 21:17:15.546502 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.546506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.553437 (+ 6931us) server_negotiation.cc:407] Connection header received
0928 21:17:15.553621 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.553625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.553677 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.553727 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.554210 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.554718 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.555816 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.556188 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.556293 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.557317 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.557320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.557323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.557351 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.558836 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.559202 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.559205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.559207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.559285 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.559513 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.559516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.559518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.559682 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.559761 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.560055 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.560174 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.254@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
I20260928 21:17:15.562707 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11371
I20260928 21:17:15.566298 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.556787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37744)
0928 21:17:15.556943 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.556956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.557041 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.557441 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.557444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.557653 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.557749 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.557756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.559772 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.559775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.560547 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.560556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.561030 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.561034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.561131 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.561601 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.561616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.563027 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.564809 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.564812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.564814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.565048 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.565317 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.565319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.565322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.565410 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.565856 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.565859 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.565949 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.566033 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.566161 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 21:17:15.566756 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.556837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.5.18.252:44759)
0928 21:17:15.557220 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.557230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.557241 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:15.557296 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.557298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.557341 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.557409 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.558135 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.559668 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.560642 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.560881 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.561004 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.563178 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.563182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.563185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.563207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.564704 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.565140 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.565144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.565146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.565237 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.565486 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.565489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.565491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.565747 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.566460 (+ 713us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.566546 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.566640 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":98,"threads_started":1}
I20260928 21:17:15.567466 11309 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.18.252:44759: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
member_type: NON_VOTER
last_known_addr {
host: "127.5.18.252"
port: 44759
}
attrs {
promote: true
}
}
dest_uuid: "e3a0ad3c30d647e68e493adb3541ef0a"
cas_config_opid_index: 5
I20260928 21:17:15.567946 11309 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } }
I20260928 21:17:15.568909 11868 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.570410 11868 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.18.252:44759
I20260928 21:17:15.570576 11868 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.570674 11868 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:15.571038 11882 master.cc:537] Master@127.5.18.252:44759 shutting down...
W20260928 21:17:15.573525 11296 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a -> Peer 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252:44759): Couldn't send request to peer 693f0f79de6a4c5393aa1fd0f10f392e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:17:15.573901 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.569035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40514)
0928 21:17:15.569358 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.569371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.569435 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.570043 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.570045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.570061 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:15.570132 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.570138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.571196 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.571200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.572343 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.572352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.573390 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.573393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.573487 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.573527 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.573716 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.573783 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260928 21:17:15.573968 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:46693
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:37051
--webserver_port=44009
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:15.574584 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.569251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.5.18.253:44807)
0928 21:17:15.569735 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.569739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.569751 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:15.569787 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.569790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.569842 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.569931 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:15.570397 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.571084 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.572443 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.573067 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.573176 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.573321 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.574402 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.574477 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260928 21:17:15.575137 11921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 LEARNER]: Refusing update from remote peer e3a0ad3c30d647e68e493adb3541ef0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:17:15.575558 11871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:17:15.577602 11921 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.579088 12058 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a0ad3c30d647e68e493adb3541ef0a. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.579176 11950 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: NON_VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: true } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.579296 11950 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: LEARNER
I20260928 21:17:15.579579 11869 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: attempting to promote NON_VOTER dc5b06321624466d94741b96c4dac7df to VOTER
I20260928 21:17:15.579937 11882 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:15.579970 11869 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } }
I20260928 21:17:15.580220 11882 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:15.580305 11882 tablet_replica.cc:330] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: stopping tablet replica
I20260928 21:17:15.581128 11921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 LEARNER]: Refusing update from remote peer e3a0ad3c30d647e68e493adb3541ef0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 21:17:15.581497 11870 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 21:17:15.581638 11296 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a -> Peer 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252:44759): Couldn't send request to peer 693f0f79de6a4c5393aa1fd0f10f392e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:17:15.579257 12058 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: LEARNER
I20260928 21:17:15.583943 11869 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc5b06321624466d94741b96c4dac7df (127.5.18.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.585227 11869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.585323 11869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:15.585448 11869 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.585510 11869 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:15.585767 11921 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc5b06321624466d94741b96c4dac7df (127.5.18.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.586459 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.546483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37734)
0928 21:17:15.546874 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.546887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.546976 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.547198 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.547201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.547389 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.547490 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.547531 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.548491 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.548494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.549278 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.549286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.549832 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.549836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.549926 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.582349 (+ 32423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.582366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.582786 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.584580 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.584583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.584585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.584923 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.585216 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.585219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.585221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.585301 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.585908 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.585910 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.586002 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.586104 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.586265 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
I20260928 21:17:15.587064 12058 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.587183 12058 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:15.587189 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.546317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.5.18.252:44759)
0928 21:17:15.546420 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.546424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.546964 (+ 540us) server_negotiation.cc:407] Connection header received
0928 21:17:15.547053 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.547056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.547096 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.547175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.547645 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.548385 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.549387 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.549729 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.549829 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.582914 (+ 33085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.582918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.582921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.582948 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.584468 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.585025 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.585028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.585030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.585116 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.585462 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.585465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.585467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.585725 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.585815 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.586953 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.587058 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:17:15.590761 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.590486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37746)
0928 21:17:15.590660 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.590711 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 21:17:15.599817 11882 master.cc:552] Master@127.5.18.252:44759 shutdown complete.
I20260928 21:17:15.599870 11882 master_runner.cc:305] Clearing existing system tablet
I20260928 21:17:15.600108 11882 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.600157 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.601354 11882 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 21:17:15.602190 12060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.602406 11882 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.602599 11882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 21:17:15.602639 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "8ff6817d00e12c1a80e76ceee7eeb35f"
server_key_iv: "26ed72ea0bf8542a874f7791ccb3a46b"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.604743 11882 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:15.607051 11882 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:17:15.607136 11882 log.cc:1196] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:17:15.607348 11882 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Deleting consensus metadata
I20260928 21:17:15.607502 11882 master_runner.cc:315] Copying system tablet from 127.5.18.254:37051
I20260928 21:17:15.608580 11882 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.18.254:37051
I20260928 21:17:15.617400 12066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.609125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:56714)
0928 21:17:15.609413 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.609428 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.609543 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.609822 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.609824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.610033 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.610163 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.610170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.611201 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.611204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.612090 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.612101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.612751 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.612755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.612853 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.613403 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.613416 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.613851 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.615833 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.615837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.615839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.616095 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.616360 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.616362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.616364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.616455 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.616850 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.616852 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.616953 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.617063 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.617202 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260928 21:17:15.617411 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.609222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.5.18.254:37051)
0928 21:17:15.609340 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.609345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.609505 (+ 160us) server_negotiation.cc:407] Connection header received
0928 21:17:15.609647 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.609650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.609701 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.609793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.610272 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.611087 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.612206 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.612555 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.612653 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.614046 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.614050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.614053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.614081 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.615706 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.616184 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.616187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.616189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.616276 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.616538 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.616541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.616543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.616761 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.616858 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.617122 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.617247 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:15.618144 11329 tablet_copy_service.cc:140] P e3a0ad3c30d647e68e493adb3541ef0a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 693f0f79de6a4c5393aa1fd0f10f392e ({username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:56714)
I20260928 21:17:15.618250 11329 tablet_copy_service.cc:161] P e3a0ad3c30d647e68e493adb3541ef0a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 693f0f79de6a4c5393aa1fd0f10f392e at {username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:56714: session id = 693f0f79de6a4c5393aa1fd0f10f392e-00000000000000000000000000000000
I20260928 21:17:15.619342 11329 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:17:15.620551 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:17:15.622959 11882 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: tablet copy: Starting download of 0 data blocks...
I20260928 21:17:15.623167 11882 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: tablet copy: Starting download of 1 WAL segments...
I20260928 21:17:15.624503 11882 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:17:15.626168 11329 tablet_copy_service.cc:342] P e3a0ad3c30d647e68e493adb3541ef0a: Request end of tablet copy session 693f0f79de6a4c5393aa1fd0f10f392e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:56714
I20260928 21:17:15.626247 11329 tablet_copy_service.cc:433] P e3a0ad3c30d647e68e493adb3541ef0a: ending tablet copy session 693f0f79de6a4c5393aa1fd0f10f392e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 693f0f79de6a4c5393aa1fd0f10f392e
I20260928 21:17:15.633288 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.635035 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.635035 12070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.635309 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.637699 11882 server_base.cc:1034] running on GCE node
I20260928 21:17:15.637849 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.638139 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.639278 11882 hybrid_clock.cc:648] HybridClock initialized: now 1790630235639259 us; error 35 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.640787 11882 init.cc:377] Logged in from keytab as oryx/127.5.18.252@KRBTEST.COM (short username oryx)
I20260928 21:17:15.641247 11882 webserver.cc:460] Webserver started at http://127.5.18.252:42737/ using document root <none> and password file <none>
I20260928 21:17:15.641419 11882 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.641458 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.642707 11882 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:15.643633 12077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.643922 11882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:15.643991 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "693f0f79de6a4c5393aa1fd0f10f392e"
format_stamp: "Formatted at 2026-09-28 21:17:15 on dist-test-slave-rc9f"
server_key: "8ff6817d00e12c1a80e76ceee7eeb35f"
server_key_iv: "26ed72ea0bf8542a874f7791ccb3a46b"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.644068 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.653985 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.655942 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.660512 11882 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.252:44759
I20260928 21:17:15.660542 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.252:44759 every 8 connection(s)
I20260928 21:17:15.661088 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
I20260928 21:17:15.662814 12130 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:17:15.663832 12130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Bootstrap starting.
I20260928 21:17:15.674327 12130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:17:15.674624 12130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Bootstrap complete.
I20260928 21:17:15.675186 12130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } }
I20260928 21:17:15.675329 12130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 693f0f79de6a4c5393aa1fd0f10f392e, State: Initialized, Role: LEARNER
I20260928 21:17:15.675431 12130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } }
I20260928 21:17:15.675755 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.675969 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: LEARNER
I20260928 21:17:15.675860 12130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:15.678956 12144 catalog_manager.cc:1259] Loaded cluster ID: 4b47dabd25294a92a45f07ccbfcdd448
I20260928 21:17:15.679077 12144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: loading cluster ID for follower catalog manager: success
I20260928 21:17:15.681366 12144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: acquiring CA information for follower catalog manager: success
I20260928 21:17:15.682713 12144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:17:15.710876 12056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:15.711098 12056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:15.711139 12056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:15.715068 12056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:15.715121 12056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:15.715183 12056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:15.720225 12056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:46693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=44009
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:15.721199 12056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:15.722199 12056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.731155 12152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.731148 12149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.731206 12056 server_base.cc:1034] running on GCE node
W20260928 21:17:15.731149 12150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.731716 12056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.732271 12056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.733421 12056 hybrid_clock.cc:648] HybridClock initialized: now 1790630235733392 us; error 44 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.735852 12056 init.cc:377] Logged in from keytab as oryx/127.5.18.193@KRBTEST.COM (short username oryx)
I20260928 21:17:15.736613 12056 webserver.cc:460] Webserver started at http://127.5.18.193:44009/ using document root <none> and password file <none>
I20260928 21:17:15.737102 12056 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.737174 12056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.740691 12056 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:15.742503 12159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.743647 12056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:15.743746 12056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "c60bbcd34a454f692bf138e5c10d48b2"
server_key_iv: "f5116a6e52368819ab8ad531f193c860"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.744283 12056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.756688 12056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.759124 12056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:15.759284 12056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.759786 12056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:15.760545 12056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:15.760588 12056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.760637 12056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:15.760665 12056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.768460 12056 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:46693
I20260928 21:17:15.768487 12272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:46693 every 8 connection(s)
I20260928 21:17:15.769346 12056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:15.775025 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12056
I20260928 21:17:15.775192 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11505
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:15.784543 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:43835
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:37051
--webserver_port=42219
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:15.785053 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.770980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:45459 (local address 127.5.18.253:44807)
0928 21:17:15.771109 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.771114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.772156 (+ 1042us) server_negotiation.cc:407] Connection header received
0928 21:17:15.773235 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.773238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.773291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.773344 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.775866 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.776629 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777696 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.777998 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.778084 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.781733 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.781736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.781739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.781766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.783321 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783804 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.783808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.783810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.783894 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.784175 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.784178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.784180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.784376 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.784451 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.784780 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.784888 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:17:15.785144 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.772751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:43415 (local address 127.5.18.254:37051)
0928 21:17:15.772889 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.772892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.772904 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:15.773199 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.773202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.773254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.773323 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.774639 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.775470 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777196 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.777548 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777645 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.781398 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.781401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.781404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.781449 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.783481 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783903 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.783906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.783908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.783984 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.784263 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.784266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.784267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.784457 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.784539 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.784850 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.784963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:17:15.785291 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.770980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:39615 (local address 127.5.18.252:44759)
0928 21:17:15.771200 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.771203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.772154 (+ 951us) server_negotiation.cc:407] Connection header received
0928 21:17:15.773234 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.773238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.773291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.773347 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.774675 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.775704 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777202 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.777551 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777648 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.780783 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.780787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.780790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.780816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.782701 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783338 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.783341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.783343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.783432 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783809 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.783812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.783814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.784050 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.784116 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.784975 (+ 859us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.785148 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260928 21:17:15.786522 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.771502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.5.18.193:45459)
0928 21:17:15.771999 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.772056 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.773012 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.773477 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.773480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.773935 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.774509 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.774521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.776757 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.776760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.777583 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.777592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.778245 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.778248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.778332 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.778806 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.778818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.781615 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.783440 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.783445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.783697 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.783989 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783991 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.783993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.784079 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.784505 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.784508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.784589 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.785242 (+ 653us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.785635 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":162,"spinlock_wait_cycles":67712,"thread_start_us":129,"threads_started":1}
I20260928 21:17:15.786541 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.771502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.5.18.193:39615)
0928 21:17:15.771999 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.772056 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.773018 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.773477 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.773480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.773938 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.774508 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.774528 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.775823 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.775826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.777074 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.777084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.777820 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.778014 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.778690 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.778706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.780608 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.782869 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.782874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.782889 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.783222 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.783535 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.783541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.783701 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.784186 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.784192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.784462 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.785786 (+ 1324us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.785920 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":103,"spinlock_wait_cycles":62720,"thread_start_us":132,"threads_started":1}
I20260928 21:17:15.787529 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.772496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:43415)
0928 21:17:15.772725 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.772736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.773018 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.773472 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.773479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.773932 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.774504 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.774519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.775665 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.775668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.777070 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.777079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.777820 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.777823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.778013 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.778637 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.778652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.781271 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.783591 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.783594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.783595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.783809 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.784089 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.784092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.784094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.784171 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.784605 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.784607 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.784687 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.785320 (+ 633us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.787386 (+ 2066us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1941,"spinlock_wait_cycles":22528,"thread_start_us":66,"threads_started":1}
I20260928 21:17:15.787652 12273 heartbeater.cc:347] Connected to a master server at 127.5.18.252:44759
I20260928 21:17:15.787859 12273 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.787962 12275 heartbeater.cc:347] Connected to a master server at 127.5.18.253:44807
I20260928 21:17:15.788084 12275 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.788136 12274 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:15.788221 12274 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.788369 12274 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:15.788370 12273 heartbeater.cc:511] Master 127.5.18.252:44759 requested a full tablet report, sending...
I20260928 21:17:15.789039 12275 heartbeater.cc:511] Master 127.5.18.253:44807 requested a full tablet report, sending...
I20260928 21:17:15.789547 11309 ts_manager.cc:194] Re-registered known tserver with Master: 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:15.789891 12094 ts_manager.cc:194] Registered new tserver with Master: 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:15.790063 11911 ts_manager.cc:194] Registered new tserver with Master: 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:15.790307 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:43415
W20260928 21:17:15.892468 12281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:15.892704 12281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:15.892745 12281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:15.896590 12281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:15.896644 12281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:15.896698 12281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:15.901664 12281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:43835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=42219
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:15.902557 12281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:15.903431 12281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:15.912710 12289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.912707 12287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:15.912796 12286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:15.912789 12281 server_base.cc:1034] running on GCE node
I20260928 21:17:15.913326 12281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:15.913928 12281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:15.915077 12281 hybrid_clock.cc:648] HybridClock initialized: now 1790630235915056 us; error 32 us; skew 500 ppm
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:15.917546 12281 init.cc:377] Logged in from keytab as oryx/127.5.18.194@KRBTEST.COM (short username oryx)
I20260928 21:17:15.918358 12281 webserver.cc:460] Webserver started at http://127.5.18.194:42219/ using document root <none> and password file <none>
I20260928 21:17:15.918857 12281 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:15.918926 12281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:15.922418 12281 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:15.924235 12296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.925350 12281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:15.925448 12281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "ffb87801d7964812bf3162738fcccb5e"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "44f45e8e12d3ad262e12de509e1bd9bb"
server_key_iv: "8920b4af3f258bb35b69d6e55b220728"
server_key_version: "encryptionkey@0"
I20260928 21:17:15.926007 12281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:15.952533 12281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:15.955354 12281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:15.955521 12281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:15.956102 12281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:15.956940 12281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:15.956985 12281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.957024 12281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:15.957044 12281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:15.964932 12281 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43835
I20260928 21:17:15.964959 12409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43835 every 8 connection(s)
I20260928 21:17:15.965870 12281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:15.971354 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12281
I20260928 21:17:15.971515 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11639
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
Sep 28 21:17:15 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630235, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
I20260928 21:17:15.978917 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:36429
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:37051
--webserver_port=39803
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:15.980239 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.967504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:38091 (local address 127.5.18.254:37051)
0928 21:17:15.967624 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.967629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.968511 (+ 882us) server_negotiation.cc:407] Connection header received
0928 21:17:15.969637 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.969640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.969686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.969748 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.970946 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.971664 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.973172 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.973383 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.973465 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.976380 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.976384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.976387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.976415 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.977960 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.978527 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.978531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.978533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.978636 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.979021 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.979024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.979026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.979241 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.979313 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.979963 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.980079 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:15.981745 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.967928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:38091)
0928 21:17:15.968369 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.968420 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.969349 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.969866 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.969873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.970263 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.970818 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.970824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.972108 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.972111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.973068 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.973077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.973501 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.973506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.973717 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.974368 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.974387 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.976178 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.978084 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.978089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.978103 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.978417 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.978738 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.978742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.978744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.978912 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.979393 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.979399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.979700 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.980380 (+ 680us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.980710 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":132,"spinlock_wait_cycles":53248,"thread_start_us":110,"threads_started":1}
I20260928 21:17:15.981894 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.968994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:54491 (local address 127.5.18.253:44807)
0928 21:17:15.969126 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.969130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.969144 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:15.969579 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.969584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.969639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.969692 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.970963 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.971763 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.974129 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.974461 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.974549 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.977543 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.977547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.977550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.977600 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.979549 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.980293 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.980296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.980298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.980381 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.980923 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.980926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.980928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.981173 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.981240 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.981637 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.981762 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:17:15.982810 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.967543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:47389 (local address 127.5.18.252:44759)
0928 21:17:15.967678 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.967683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.968514 (+ 831us) server_negotiation.cc:407] Connection header received
0928 21:17:15.969535 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.969538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.969629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.969698 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:15.970936 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.971757 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.974582 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.974909 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.974975 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.977862 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:15.977866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:15.977869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:15.977896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:15.979488 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.979929 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.979932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.979934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.980014 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:15.980305 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:15.980308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:15.980310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:15.980541 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:15.980611 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.982531 (+ 1920us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.982661 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:17:15.982802 12411 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:15.983016 12411 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.983453 12411 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:15.984087 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.968846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.5.18.194:54491)
0928 21:17:15.969088 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.969099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.969357 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.969866 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.969873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.970263 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.970816 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.970824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.973047 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.973050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.973965 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.973975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.974658 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.974662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.974748 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.975379 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.975393 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.977375 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.979947 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.979950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.979952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.980184 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.980742 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.980745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.980747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.980827 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.981353 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.981355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.981436 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.982404 (+ 968us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.982532 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":72,"spinlock_wait_cycles":8832,"thread_start_us":78,"threads_started":1}
I20260928 21:17:15.984464 11309 ts_manager.cc:194] Re-registered known tserver with Master: ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835)
I20260928 21:17:15.984647 12412 heartbeater.cc:347] Connected to a master server at 127.5.18.253:44807
I20260928 21:17:15.984726 12412 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.984871 12412 heartbeater.cc:511] Master 127.5.18.253:44807 requested a full tablet report, sending...
I20260928 21:17:15.985172 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:38091
I20260928 21:17:15.985500 11911 ts_manager.cc:194] Registered new tserver with Master: ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835)
I20260928 21:17:15.987988 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.981215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37750)
0928 21:17:15.981463 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.981476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.981559 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.983342 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.983346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.983364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:15.983447 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.983452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.985670 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.985674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.986815 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.986827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.987725 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.987728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.987792 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.987809 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.987850 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.987901 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 21:17:15.988071 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.981721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.5.18.252:44759)
0928 21:17:15.982976 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:15.982980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:15.982993 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:15.983032 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:15.983035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:15.983089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:15.983175 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:15.983578 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.985493 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.986967 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.987598 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.987702 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.987851 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:15.987887 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 21:17:15.987948 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":52,"threads_started":1}
I20260928 21:17:15.988646 12104 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 21:17:15.988705 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:15.967927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.5.18.194:47389)
0928 21:17:15.968369 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:15.968420 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:15.969346 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:15.969864 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:15.969871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:15.970262 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:15.970802 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.970814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.973659 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.973662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.974441 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:15.974451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:15.975124 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:15.975127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:15.975221 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:15.975574 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:15.975588 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:15.977726 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:15.979612 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.979615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.979617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.979830 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.980124 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:15.980126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:15.980128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:15.980212 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:15.981897 (+ 1685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:15.981900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:15.981984 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 21:17:15.988375 (+ 6391us) client_negotiation.cc:240] Negotiation successful
0928 21:17:15.988539 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":55,"spinlock_wait_cycles":21760,"thread_start_us":102,"threads_started":1}
I20260928 21:17:15.989179 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a0ad3c30d647e68e493adb3541ef0a. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:15.989300 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: LEARNER
I20260928 21:17:15.989364 12410 heartbeater.cc:347] Connected to a master server at 127.5.18.252:44759
I20260928 21:17:15.989452 12410 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:15.989665 12410 heartbeater.cc:511] Master 127.5.18.252:44759 requested a full tablet report, sending...
I20260928 21:17:15.990231 12094 ts_manager.cc:194] Registered new tserver with Master: ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835)
W20260928 21:17:16.091290 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:16.091534 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:16.091607 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:16.095536 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:16.095588 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:16.095661 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:16.100768 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:36429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=39803
--tserver_master_addrs=127.5.18.254:37051,127.5.18.253:44807,127.5.18.252:44759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:16.101771 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:16.102730 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:16.111799 12426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:16.111804 12423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:16.112234 12424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:16.112282 12417 server_base.cc:1034] running on GCE node
I20260928 21:17:16.112687 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:16.113396 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:16.114547 12417 hybrid_clock.cc:648] HybridClock initialized: now 1790630236114534 us; error 56 us; skew 500 ppm
Sep 28 21:17:16 dist-test-slave-rc9f krb5kdc[11264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630236, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:16.117156 12417 init.cc:377] Logged in from keytab as oryx/127.5.18.195@KRBTEST.COM (short username oryx)
I20260928 21:17:16.118012 12417 webserver.cc:460] Webserver started at http://127.5.18.195:39803/ using document root <none> and password file <none>
I20260928 21:17:16.118513 12417 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:16.118587 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:16.122359 12417 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:16.124189 12433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:16.125356 12417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:16.125458 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "57163cb0239e46be85b86e8c04bb7dc9"
format_stamp: "Formatted at 2026-09-28 21:17:14 on dist-test-slave-rc9f"
server_key: "cf469e979627da0a1baec84c41df9278"
server_key_iv: "31493d1f4553cf7c421e9c52109e9e87"
server_key_version: "encryptionkey@0"
I20260928 21:17:16.126027 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:16.144241 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:16.147606 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:16.147781 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:16.148355 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:16.149204 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:16.149248 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:16.149290 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:16.149317 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:16.157385 12417 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:36429
I20260928 21:17:16.157452 12546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:36429 every 8 connection(s)
I20260928 21:17:16.158370 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:16.165179 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12417
Sep 28 21:17:16 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630236, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.253@KRBTEST.COM
Sep 28 21:17:16 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630236, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
Sep 28 21:17:16 dist-test-slave-rc9f krb5kdc[11264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630236, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.252@KRBTEST.COM
I20260928 21:17:16.174577 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.161364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.5.18.195:56601)
0928 21:17:16.161624 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.161636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.161950 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.162448 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.162455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.162931 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:16.163475 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.163481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164259 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.164263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.165428 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.165437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.166173 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.166177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.166523 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.167225 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:16.167245 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:16.169093 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:16.171107 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.171113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.171126 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.171635 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.171933 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.171936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.171938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.172109 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.172546 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:16.172553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:16.172840 (+ 287us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.173221 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.173526 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":208,"spinlock_wait_cycles":15744,"thread_start_us":73,"threads_started":1}
I20260928 21:17:16.175321 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.161473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56601 (local address 127.5.18.253:44807)
0928 21:17:16.161617 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.161622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.161705 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:17:16.162148 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.162151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.162200 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.162254 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.163618 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164133 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.165581 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.165916 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.165988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.169237 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.169241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.169244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.169271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.171019 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.171741 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.171744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.171746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.171832 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.172213 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.172216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.172218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.172411 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.172495 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.175070 (+ 2575us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.175181 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:17:16.175794 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.160420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:53979)
0928 21:17:16.160861 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.160913 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.161938 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.162447 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.162454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.162945 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:16.163461 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.163473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164561 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.164564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.165426 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.165436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.166182 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.166186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.166525 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.167317 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:16.167329 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:16.171395 (+ 4066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:16.173722 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.173725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.173727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.173963 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.174479 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.174484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.174574 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.175031 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:16.175034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:16.175128 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.175239 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.175359 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":165,"spinlock_wait_cycles":13952,"thread_start_us":98,"threads_started":1}
I20260928 21:17:16.176240 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.159983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:53979 (local address 127.5.18.254:37051)
0928 21:17:16.160105 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.160110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.161012 (+ 902us) server_negotiation.cc:407] Connection header received
0928 21:17:16.162139 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.162142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.162199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.162266 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.163629 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164414 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.165607 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.165936 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.166015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.171547 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.171551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.171554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.171595 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.173611 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174285 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.174289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.174291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.174378 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174673 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.174676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.174678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.174880 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.174972 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.175975 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.176095 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:17:16.176297 12549 heartbeater.cc:347] Connected to a master server at 127.5.18.253:44807
I20260928 21:17:16.176545 12549 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:16.176973 12548 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37051
I20260928 21:17:16.177066 12548 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:16.177083 12549 heartbeater.cc:511] Master 127.5.18.253:44807 requested a full tablet report, sending...
I20260928 21:17:16.177224 12548 heartbeater.cc:511] Master 127.5.18.254:37051 requested a full tablet report, sending...
I20260928 21:17:16.178344 11911 ts_manager.cc:194] Registered new tserver with Master: 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429)
I20260928 21:17:16.178799 11309 ts_manager.cc:194] Re-registered known tserver with Master: 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429)
I20260928 21:17:16.179183 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.160017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:32921 (local address 127.5.18.252:44759)
0928 21:17:16.160148 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.160153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.161008 (+ 855us) server_negotiation.cc:407] Connection header received
0928 21:17:16.162148 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.162151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.162200 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.162257 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.163602 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164420 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.165627 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.165965 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.166027 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.171902 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.171906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.171908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.171932 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.174013 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174473 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.174476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.174477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.174561 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.175085 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.175088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.175090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.175277 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.175352 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.178912 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.179039 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:17:16.179430 11309 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:53979
I20260928 21:17:16.179481 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.167364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.5.18.252:44759)
0928 21:17:16.167475 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.167478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.168098 (+ 620us) server_negotiation.cc:407] Connection header received
0928 21:17:16.168270 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.168273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.168321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.168374 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.169945 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.170700 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.172900 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.173194 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.173274 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.176116 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.176120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.176122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.176144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.177767 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.178204 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.178208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.178210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.178290 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.178612 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.178616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.178617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.178817 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.178911 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.179236 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.179336 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:17:16.179589 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.160420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.5.18.195:32921)
0928 21:17:16.160862 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.160913 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.161950 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.162448 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.162456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.162931 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:16.163472 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.163481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.164520 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.164523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.165489 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.165496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.166173 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.166177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.168167 (+ 1990us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.168614 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:16.168628 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:16.171767 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:16.174122 (+ 2355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.174128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.174347 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.174896 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:16.174899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:16.174900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:16.174988 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:16.178656 (+ 3668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:16.178659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:16.178744 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.179305 (+ 561us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.179430 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":199,"spinlock_wait_cycles":32128,"thread_start_us":120,"threads_started":1}
I20260928 21:17:16.180641 12547 heartbeater.cc:347] Connected to a master server at 127.5.18.252:44759
I20260928 21:17:16.180742 12547 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:16.181151 12547 heartbeater.cc:511] Master 127.5.18.252:44759 requested a full tablet report, sending...
I20260928 21:17:16.181770 12094 ts_manager.cc:194] Registered new tserver with Master: 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429)
I20260928 21:17:16.183584 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.167555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.5.18.253:44807)
0928 21:17:16.167943 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.167947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.169133 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 21:17:16.169174 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.169177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.169225 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.169303 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.170662 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.171431 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.174131 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.174457 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.174553 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.178369 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.178373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.178376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.178398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.181995 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.182426 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.182428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.182430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.182509 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.182853 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.182856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.182858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.183022 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.183124 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.183340 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.183452 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260928 21:17:16.184155 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.166939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.5.18.254:37051)
0928 21:17:16.167380 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.167385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.168178 (+ 793us) server_negotiation.cc:407] Connection header received
0928 21:17:16.168357 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.168361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.168411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.168472 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.169917 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.170694 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.175565 (+ 4871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.175904 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.175972 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.180991 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.180995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.180997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.181022 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.182704 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.183072 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.183076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.183077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.183152 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.183384 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.183387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.183389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.183585 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.183696 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.183936 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.184029 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":76,"threads_started":1}
I20260928 21:17:16.186273 11309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:16.189061 11309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:16.191236 11868 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } }
I20260928 21:17:16.191351 11868 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:16.194478 11868 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: attempting to promote NON_VOTER 693f0f79de6a4c5393aa1fd0f10f392e to VOTER
I20260928 21:17:16.194792 11868 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } }
I20260928 21:17:16.195791 12104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 LEARNER]: Refusing update from remote peer e3a0ad3c30d647e68e493adb3541ef0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:17:16.196147 12566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:17:16.197392 11921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Refusing update from remote peer e3a0ad3c30d647e68e493adb3541ef0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:17:16.197623 12566 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: NON_VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: true } } } pending_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.197728 12565 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:17:16.198529 12565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.199251 12104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.199747 11921 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.199951 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.200048 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:16.200237 12565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.200309 12565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:16.197744 12566 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:16.200903 12133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.200999 12133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:16.201759 12574 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:16.201858 12574 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:16.206563 11866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.201913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:36429 (local address 127.0.0.1:45088)
0928 21:17:16.202057 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.202072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.202157 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.202791 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.202794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.202812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.202907 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.202913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.204348 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.204350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.205067 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.205073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.206217 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.206220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.206305 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.206327 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.206378 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.206443 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 21:17:16.206643 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.202047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.5.18.195:36429)
0928 21:17:16.202312 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.202322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.202343 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:17:16.202417 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.202423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.202633 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.203068 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.203205 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.204265 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.205291 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.206089 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.206223 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.206373 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.206423 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.206507 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
I20260928 21:17:16.207563 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.202419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:46693 (local address 127.0.0.1:58358)
0928 21:17:16.203162 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.203174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.203248 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.203869 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.203871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.203881 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.203946 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.203950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.204919 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.204923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.206041 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.206050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.207197 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.207200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.207287 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.207308 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.207385 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.207447 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":98,"threads_started":1}
I20260928 21:17:16.207825 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.202703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.5.18.193:46693)
0928 21:17:16.203450 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.203461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.203482 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:17:16.203564 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.203569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.203747 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.203889 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.204000 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.204815 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.206159 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.206924 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.207018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.207141 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.207451 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.207542 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":103,"threads_started":1}
I20260928 21:17:16.209764 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.202424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43835 (local address 127.0.0.1:38266)
0928 21:17:16.203386 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.203395 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.203443 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.205627 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.205630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.205646 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.205720 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.205725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.206966 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.206969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.207872 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.207883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.209381 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.209385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.209468 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.209491 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.209584 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.209649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":181,"thread_start_us":69,"threads_started":1}
I20260928 21:17:16.209837 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.203054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.5.18.194:43835)
0928 21:17:16.205069 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.205079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.205099 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:17:16.205165 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.205170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.205343 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.205472 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.205827 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.206860 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.208508 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.209255 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.209359 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.209481 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.209619 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.209706 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1919,"thread_start_us":104,"threads_started":1}
I20260928 21:17:16.209613 12481 tablet_service.cc:1467] Processing CreateTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 (DEFAULT_TABLE table=test-table [id=c8f11801c525499f9cf0f53d69aa0732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:16.210602 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234f77ac25e14d8696a2bcaff22c0cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:16.210413 12207 tablet_service.cc:1467] Processing CreateTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 (DEFAULT_TABLE table=test-table [id=c8f11801c525499f9cf0f53d69aa0732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:16.211314 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234f77ac25e14d8696a2bcaff22c0cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:16.212579 12344 tablet_service.cc:1467] Processing CreateTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 (DEFAULT_TABLE table=test-table [id=c8f11801c525499f9cf0f53d69aa0732]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:16.213508 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234f77ac25e14d8696a2bcaff22c0cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:16.217691 12580 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Bootstrap starting.
I20260928 21:17:16.218755 12581 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Bootstrap starting.
I20260928 21:17:16.219686 12580 tablet_bootstrap.cc:654] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:16.220357 12580 log.cc:824] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:16.220666 12582 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Bootstrap starting.
I20260928 21:17:16.220780 12581 tablet_bootstrap.cc:654] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:16.221468 12581 log.cc:824] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:16.222565 12580 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: No bootstrap required, opened a new log
I20260928 21:17:16.222739 12580 ts_tablet_manager.cc:1397] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:17:16.222757 12582 tablet_bootstrap.cc:654] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:16.223304 12581 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: No bootstrap required, opened a new log
I20260928 21:17:16.223415 12582 log.cc:824] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:16.223474 12581 ts_tablet_manager.cc:1397] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:17:16.224830 12582 tablet_bootstrap.cc:492] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: No bootstrap required, opened a new log
I20260928 21:17:16.224983 12582 ts_tablet_manager.cc:1397] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 21:17:16.227115 12580 raft_consensus.cc:348] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.227376 12580 raft_consensus.cc:374] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:16.227430 12580 raft_consensus.cc:729] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827fc91bd3704b08b1f5b9663c4d94d7, State: Initialized, Role: FOLLOWER
I20260928 21:17:16.227562 12581 raft_consensus.cc:348] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.227788 12581 raft_consensus.cc:374] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:16.227835 12581 raft_consensus.cc:729] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57163cb0239e46be85b86e8c04bb7dc9, State: Initialized, Role: FOLLOWER
I20260928 21:17:16.227880 12580 consensus_queue.cc:260] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.228272 12581 consensus_queue.cc:260] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.228591 12582 raft_consensus.cc:348] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.228856 12580 ts_tablet_manager.cc:1428] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:17:16.228863 12582 raft_consensus.cc:374] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:16.228910 12582 raft_consensus.cc:729] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb87801d7964812bf3162738fcccb5e, State: Initialized, Role: FOLLOWER
I20260928 21:17:16.229027 12274 heartbeater.cc:503] Master 127.5.18.254:37051 was elected leader, sending a full tablet report...
I20260928 21:17:16.229161 12581 ts_tablet_manager.cc:1428] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:17:16.229442 12548 heartbeater.cc:503] Master 127.5.18.254:37051 was elected leader, sending a full tablet report...
I20260928 21:17:16.229272 12582 consensus_queue.cc:260] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.231532 12582 ts_tablet_manager.cc:1428] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:17:16.232633 12411 heartbeater.cc:503] Master 127.5.18.254:37051 was elected leader, sending a full tablet report...
W20260928 21:17:16.271119 12276 tablet.cc:2369] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:16.359895 12587 raft_consensus.cc:482] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:16.360122 12587 raft_consensus.cc:504] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.361850 12587 leader_election.cc:288] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835), 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:16.364316 12586 raft_consensus.cc:482] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:16.364495 12586 raft_consensus.cc:504] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.365615 12586 leader_election.cc:288] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429), ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835)
I20260928 21:17:16.367548 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.362277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43835 (local address 127.5.18.195:33973)
0928 21:17:16.362439 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.362452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.362548 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.362883 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.362885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.362901 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.362996 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.363002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.363962 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.363965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.365193 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.365202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.367130 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.367134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.367236 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.367260 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.367325 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.367395 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 21:17:16.367951 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.361816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:33973 (local address 127.5.18.194:43835)
0928 21:17:16.361932 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.361936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.362627 (+ 691us) server_negotiation.cc:407] Connection header received
0928 21:17:16.362682 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.362685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.362732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.362788 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.363098 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.363845 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.365334 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.365914 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.366017 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.366068 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.367772 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.367842 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:17:16.368219 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.361727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:46693 (local address 127.5.18.195:57615)
0928 21:17:16.362098 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.362114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.362207 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.362600 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.362603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.362619 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.362742 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.362748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.363755 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.363758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.366660 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.366671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.367807 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.367811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.367917 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.367943 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.368015 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.368087 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":1670}
I20260928 21:17:16.368480 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb87801d7964812bf3162738fcccb5e" is_pre_election: true
I20260928 21:17:16.368755 12364 raft_consensus.cc:2454] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 0.
I20260928 21:17:16.368912 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.362023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:57615 (local address 127.5.18.193:46693)
0928 21:17:16.362132 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.362136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.362153 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:17:16.362358 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.362362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.362416 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.362482 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.362864 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.363618 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.366810 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.367429 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.367536 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.367585 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.368720 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.368796 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:16.369184 12435 leader_election.cc:302] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57163cb0239e46be85b86e8c04bb7dc9, ffb87801d7964812bf3162738fcccb5e; no voters:
I20260928 21:17:16.369423 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" is_pre_election: true
I20260928 21:17:16.369619 12227 raft_consensus.cc:2454] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 0.
I20260928 21:17:16.370364 12587 raft_consensus.cc:2781] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:16.370491 12587 raft_consensus.cc:482] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:16.370580 12587 raft_consensus.cc:3037] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:16.371954 12587 raft_consensus.cc:504] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.372649 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb87801d7964812bf3162738fcccb5e"
I20260928 21:17:16.372773 12364 raft_consensus.cc:3037] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:16.373445 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.366327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:47647 (local address 127.5.18.195:36429)
0928 21:17:16.366420 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.366423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.369601 (+ 3178us) server_negotiation.cc:407] Connection header received
0928 21:17:16.369748 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.369751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.369786 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.369823 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.370097 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.370832 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.372288 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.372865 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.372941 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.372981 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.373282 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.373344 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:16.373811 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.366277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:36429 (local address 127.5.18.193:47647)
0928 21:17:16.369506 (+ 3229us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.369531 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.369660 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.369919 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.369922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.369935 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.370007 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.370012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.370948 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.370951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.372159 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.372169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.373065 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.373068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.373166 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.373187 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.373616 (+ 429us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.373687 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3167,"thread_start_us":79,"threads_started":1}
I20260928 21:17:16.374121 12364 raft_consensus.cc:2454] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 1.
I20260928 21:17:16.374431 12435 leader_election.cc:302] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57163cb0239e46be85b86e8c04bb7dc9, ffb87801d7964812bf3162738fcccb5e; no voters:
I20260928 21:17:16.374603 12587 leader_election.cc:288] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 election: Requested vote from peers ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835), 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:16.374722 12587 raft_consensus.cc:2781] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:16.374810 12587 raft_consensus.cc:686] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 57163cb0239e46be85b86e8c04bb7dc9, State: Running, Role: LEADER
I20260928 21:17:16.375102 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
I20260928 21:17:16.375217 12227 raft_consensus.cc:3037] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:16.374902 12501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57163cb0239e46be85b86e8c04bb7dc9" is_pre_election: true
I20260928 21:17:16.376123 12587 consensus_queue.cc:237] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } }
I20260928 21:17:16.376293 12227 raft_consensus.cc:2454] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 1.
I20260928 21:17:16.376919 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.366630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43835 (local address 127.5.18.193:44227)
0928 21:17:16.373272 (+ 6642us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:16.373287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:16.373421 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:16.373783 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:16.373787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:16.373806 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:16.373917 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.373924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.374823 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.374826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.375667 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:16.375676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.376531 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.376535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:16.376616 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.376637 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:16.376728 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:17:16.376789 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6574,"thread_start_us":49,"threads_started":1}
I20260928 21:17:16.376961 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.366866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:44227 (local address 127.5.18.194:43835)
0928 21:17:16.368083 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.368086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.373355 (+ 5269us) server_negotiation.cc:407] Connection header received
0928 21:17:16.373533 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.373536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.373600 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.373650 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:16.374082 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.374705 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.375779 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.376312 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.376417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.376459 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.376760 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.376823 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":92,"threads_started":1}
I20260928 21:17:16.377291 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "234f77ac25e14d8696a2bcaff22c0cb0" candidate_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb87801d7964812bf3162738fcccb5e" is_pre_election: true
I20260928 21:17:16.377506 12364 raft_consensus.cc:2379] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 827fc91bd3704b08b1f5b9663c4d94d7 in current term 1: Already voted for candidate 57163cb0239e46be85b86e8c04bb7dc9 in this term.
I20260928 21:17:16.377948 12161 leader_election.cc:302] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 827fc91bd3704b08b1f5b9663c4d94d7; no voters: 57163cb0239e46be85b86e8c04bb7dc9, ffb87801d7964812bf3162738fcccb5e
I20260928 21:17:16.378396 12586 raft_consensus.cc:2726] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 21:17:16.379745 11307 catalog_manager.cc:5495] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "57163cb0239e46be85b86e8c04bb7dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } health_report { overall_health: UNKNOWN } } }
I20260928 21:17:16.395404 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.392450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.5.18.195:36429)
0928 21:17:16.392565 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.392568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.392579 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:16.392703 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.392705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.392752 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.392831 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:16.393012 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.393484 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.394276 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.394479 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.394543 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.394744 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:16.395133 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:16.395217 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.395242 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.395283 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:17:16.400627 12364 raft_consensus.cc:1261] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Refusing update from remote peer 57163cb0239e46be85b86e8c04bb7dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:16.400667 12227 raft_consensus.cc:1261] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Refusing update from remote peer 57163cb0239e46be85b86e8c04bb7dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:16.401278 12592 consensus_queue.cc:1035] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:16.401446 12587 consensus_queue.cc:1035] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:16.411860 12597 mvcc.cc:205] Tried to move back new op lower bound from 7334421448290455552 to 7334421448199966720. Current Snapshot: MvccSnapshot[applied={T|T < 7334421448290455552}]
I20260928 21:17:16.414908 12596 mvcc.cc:205] Tried to move back new op lower bound from 7334421448290455552 to 7334421448199966720. Current Snapshot: MvccSnapshot[applied={T|T < 7334421448290455552}]
I20260928 21:17:16.418131 11307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:16.418279 11307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:16.421478 11307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3a0ad3c30d647e68e493adb3541ef0a: Sending DeleteTablet for 3 replicas of tablet 234f77ac25e14d8696a2bcaff22c0cb0
I20260928 21:17:16.422271 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:16 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:58358
I20260928 21:17:16.422494 12481 tablet_service.cc:1514] Processing DeleteTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:16 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:45088
I20260928 21:17:16.422884 12606 tablet_replica.cc:330] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: stopping tablet replica
I20260928 21:17:16.422938 12344 tablet_service.cc:1514] Processing DeleteTablet for tablet 234f77ac25e14d8696a2bcaff22c0cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:16 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:38266
I20260928 21:17:16.423301 12606 raft_consensus.cc:2229] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:16.423686 12606 raft_consensus.cc:2258] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:16.424324 12605 tablet_replica.cc:330] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: stopping tablet replica
I20260928 21:17:16.424610 12605 raft_consensus.cc:2229] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:16.424897 12605 raft_consensus.cc:2258] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:16.425302 12606 ts_tablet_manager.cc:1905] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:16.426798 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11277
I20260928 21:17:16.426826 12605 ts_tablet_manager.cc:1905] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:16.428007 12606 ts_tablet_manager.cc:1918] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:16.428122 12606 log.cc:1196] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/234f77ac25e14d8696a2bcaff22c0cb0
I20260928 21:17:16.428442 12606 ts_tablet_manager.cc:1939] T 234f77ac25e14d8696a2bcaff22c0cb0 P 57163cb0239e46be85b86e8c04bb7dc9: Deleting consensus metadata
I20260928 21:17:16.429663 12605 ts_tablet_manager.cc:1918] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:16.429749 12605 log.cc:1196] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/234f77ac25e14d8696a2bcaff22c0cb0
I20260928 21:17:16.430078 12605 ts_tablet_manager.cc:1939] T 234f77ac25e14d8696a2bcaff22c0cb0 P 827fc91bd3704b08b1f5b9663c4d94d7: Deleting consensus metadata
W20260928 21:17:16.434082 12435 connection.cc:535] server connection from 127.0.0.1:45088 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:17:16.434254 12162 connection.cc:535] server connection from 127.0.0.1:58358 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:17:16.434732 12607 tablet_replica.cc:330] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: stopping tablet replica
I20260928 21:17:16.435102 12607 raft_consensus.cc:2229] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:16.435452 12607 raft_consensus.cc:2258] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:16.436926 12607 ts_tablet_manager.cc:1905] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:16.442118 12607 ts_tablet_manager.cc:1918] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:16.442289 12607 log.cc:1196] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/234f77ac25e14d8696a2bcaff22c0cb0
I20260928 21:17:16.443544 12607 ts_tablet_manager.cc:1939] T 234f77ac25e14d8696a2bcaff22c0cb0 P ffb87801d7964812bf3162738fcccb5e: Deleting consensus metadata
I20260928 21:17:16.443773 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.425732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.5.18.253:44807)
0928 21:17:16.425836 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.425840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.425852 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:16.425886 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.425889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.425932 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.426014 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.434479 (+ 8465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.435285 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.436409 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.436767 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.436852 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.438633 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.438637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.438640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.438664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.440249 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.440701 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.440704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.440706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.440783 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.441038 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.441041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.441043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.441232 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.441343 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.443252 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.443378 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 21:17:16.445072 12299 connection.cc:441] server connection from 127.0.0.1:38266 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38266 (request call id 1) could send its response
I20260928 21:17:16.448217 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.437137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.5.18.252:44759)
0928 21:17:16.437244 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.437248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.439403 (+ 2155us) server_negotiation.cc:407] Connection header received
0928 21:17:16.439504 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.439507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.439565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.439632 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.440883 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.441639 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.442673 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.443041 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.443115 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.444865 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.444869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.444872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.444897 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.446428 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.446973 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.446976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.446978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.447053 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.447328 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.447331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.447333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.447528 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.447619 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.447952 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.448074 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:16.448217 12033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.436446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.5.18.253:44807)
0928 21:17:16.436799 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.436803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.438240 (+ 1437us) server_negotiation.cc:407] Connection header received
0928 21:17:16.438399 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.438402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.438445 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.438492 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.439532 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.440255 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.441307 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.441638 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.441708 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.444733 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.444737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.444740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.444762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.446353 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.446929 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.446932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.446934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.447026 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.447328 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.447331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.447333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.447517 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.447636 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.447952 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.448069 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260928 21:17:16.453825 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:16.426329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.5.18.252:44759)
0928 21:17:16.426440 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:16.426444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:16.442027 (+ 15583us) server_negotiation.cc:407] Connection header received
0928 21:17:16.442435 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:16.442439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:16.442483 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:16.442530 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:16.444552 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.446046 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.447594 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:16.447952 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:16.448020 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:16.450099 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:16.450102 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:16.450103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:16.450124 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:16.451738 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.452214 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.452217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.452219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.452325 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:16.452600 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:16.452603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:16.452605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:16.452873 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:16.453042 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:16.453238 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:17:16.453528 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:17.416656 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.416218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:51567)
0928 21:17:17.416434 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.416544 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
W20260928 21:17:17.416987 12411 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:37051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.417239 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.416827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:39149)
0928 21:17:17.417058 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.417143 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
W20260928 21:17:17.417515 12274 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:37051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.417783 12618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.417374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:36041)
0928 21:17:17.417610 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.417694 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
W20260928 21:17:17.418021 12548 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:37051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.922319 12619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0ad3c30d647e68e493adb3541ef0a)
I20260928 21:17:17.922538 12619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } }
I20260928 21:17:17.923470 12619 leader_election.cc:288] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051), dc5b06321624466d94741b96c4dac7df (127.5.18.253:44807)
I20260928 21:17:17.924072 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.923636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57178)
0928 21:17:17.923910 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.923973 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":53,"threads_started":1}
W20260928 21:17:17.924226 12079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.924279 12622 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3a0ad3c30d647e68e493adb3541ef0a)
I20260928 21:17:17.924443 12622 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } }
I20260928 21:17:17.925077 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.924864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57188)
0928 21:17:17.924967 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.925017 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:17:17.925191 12079 leader_election.cc:334] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051): Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.925415 12622 leader_election.cc:288] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051), 693f0f79de6a4c5393aa1fd0f10f392e (127.5.18.252:44759)
I20260928 21:17:17.925922 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.925485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57200)
0928 21:17:17.925767 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.925832 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":148,"threads_started":1}
W20260928 21:17:17.926301 11896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.927170 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.926860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57204)
0928 21:17:17.927068 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.927111 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 21:17:17.927269 11896 leader_election.cc:334] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051): Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.928660 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.924096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.5.18.253:44807)
0928 21:17:17.924443 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:17.924447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:17.924677 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:17:17.924822 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:17.924826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:17.924878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:17.924963 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:17.925408 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.926187 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.927657 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.928190 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.928265 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.928311 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:17.928503 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:17:17.928556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":52,"threads_started":1}
I20260928 21:17:17.928689 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.923969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:44807 (local address 127.0.0.1:40562)
0928 21:17:17.924591 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.924604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:17.924710 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:17.924990 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:17.924993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:17.925020 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:17.925114 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.925120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.926293 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.926296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.927553 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.927562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.928297 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.928300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.928391 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.928417 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:17:17.928493 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 21:17:17.928559 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
I20260928 21:17:17.929214 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc5b06321624466d94741b96c4dac7df" is_pre_election: true
I20260928 21:17:17.929391 11921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 693f0f79de6a4c5393aa1fd0f10f392e in term 1.
I20260928 21:17:17.929774 12079 leader_election.cc:302] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 693f0f79de6a4c5393aa1fd0f10f392e, dc5b06321624466d94741b96c4dac7df; no voters: e3a0ad3c30d647e68e493adb3541ef0a
I20260928 21:17:17.929989 12619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:17:17.930074 12619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3a0ad3c30d647e68e493adb3541ef0a)
I20260928 21:17:17.930121 12619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:17:17.930577 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.925530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44759 (local address 127.0.0.1:37790)
0928 21:17:17.925852 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.925866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:17.925949 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:17.926946 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:17.926949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:17.926965 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:17.927044 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.927050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.927920 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.927922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.929036 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.929046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.929962 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.929967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.930297 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.930318 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:17.930386 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:17:17.930457 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":60,"threads_started":1}
I20260928 21:17:17.930625 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.925761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.5.18.252:44759)
0928 21:17:17.926684 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:17.926695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:17.926709 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:17.926746 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:17.926749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:17.926786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:17.926860 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:17.927150 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.927828 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.929136 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.929711 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.929803 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.929855 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:17.930445 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 21:17:17.930524 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260928 21:17:17.930936 12104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc5b06321624466d94741b96c4dac7df" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" is_pre_election: true
I20260928 21:17:17.931249 12619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } }
I20260928 21:17:17.931460 12104 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc5b06321624466d94741b96c4dac7df in current term 2: Already voted for candidate 693f0f79de6a4c5393aa1fd0f10f392e in this term.
I20260928 21:17:17.931529 12619 leader_election.cc:288] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 election: Requested vote from peers e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051), dc5b06321624466d94741b96c4dac7df (127.5.18.253:44807)
I20260928 21:17:17.931880 11898 leader_election.cc:302] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc5b06321624466d94741b96c4dac7df; no voters: 693f0f79de6a4c5393aa1fd0f10f392e, e3a0ad3c30d647e68e493adb3541ef0a
I20260928 21:17:17.932085 12622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:17:17.932367 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc5b06321624466d94741b96c4dac7df"
I20260928 21:17:17.932435 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.931671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57208)
0928 21:17:17.932345 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.932387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617}
I20260928 21:17:17.932964 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.932788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57218)
0928 21:17:17.932874 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.932917 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:17:17.933038 12622 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 21:17:17.933095 12079 leader_election.cc:334] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051): Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
I20260928 21:17:17.934053 11921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 693f0f79de6a4c5393aa1fd0f10f392e in term 2.
I20260928 21:17:17.934307 12079 leader_election.cc:302] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 693f0f79de6a4c5393aa1fd0f10f392e, dc5b06321624466d94741b96c4dac7df; no voters: e3a0ad3c30d647e68e493adb3541ef0a
I20260928 21:17:17.934492 12619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:17:17.934653 12619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [term 2 LEADER]: Becoming Leader. State: Replica: 693f0f79de6a4c5393aa1fd0f10f392e, State: Running, Role: LEADER
I20260928 21:17:17.934857 12619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } }
I20260928 21:17:17.935998 12629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693f0f79de6a4c5393aa1fd0f10f392e. Latest consensus state: current_term: 2 leader_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:17.936089 12629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:17.936304 12631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:17.938416 12631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:17.938791 12631 catalog_manager.cc:1259] Loaded cluster ID: 4b47dabd25294a92a45f07ccbfcdd448
I20260928 21:17:17.938836 12631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:17.939343 12631 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:17.939708 12631 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Loaded TSK: 0
I20260928 21:17:17.940326 12631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:17.952607 12092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:17.955704 12092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:17.958683 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.958453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57236)
0928 21:17:17.958564 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.958616 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:17:17.958698 11921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [term 2 FOLLOWER]: Refusing update from remote peer 693f0f79de6a4c5393aa1fd0f10f392e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 21:17:17.959168 12629 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 21:17:17.959429 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.959223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.0.0.1:57238)
0928 21:17:17.959318 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.959368 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 21:17:17.959591 12079 consensus_peers.cc:487] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e -> Peer e3a0ad3c30d647e68e493adb3541ef0a (127.5.18.254:37051): Couldn't send request to peer e3a0ad3c30d647e68e493adb3541ef0a. Status: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:17:17.960455 12622 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 693f0f79de6a4c5393aa1fd0f10f392e. Latest consensus state: current_term: 2 leader_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:17.960572 12622 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:17.961678 12629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:17.961894 12629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:17.962131 12622 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e3a0ad3c30d647e68e493adb3541ef0a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37051 } } peers { permanent_uuid: "dc5b06321624466d94741b96c4dac7df" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 44807 } attrs { promote: false } } peers { permanent_uuid: "693f0f79de6a4c5393aa1fd0f10f392e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44759 } attrs { promote: false } } }
I20260928 21:17:17.962251 12622 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5b06321624466d94741b96c4dac7df [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:17:17.971077 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.967260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43835 (local address 127.0.0.1:38280)
0928 21:17:17.967365 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.967379 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:17.967452 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:17.967833 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:17.967835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:17.967853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:17.967953 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.967959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.968923 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.968925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.969837 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.969845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.970725 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.970729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.970815 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.970842 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:17:17.970891 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:17:17.970961 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 21:17:17.971589 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.967365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.5.18.194:43835)
0928 21:17:17.967592 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:17.967597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:17.967612 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:17:17.967661 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:17.967663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:17.967706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:17.967818 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:17.968075 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.968811 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.969973 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.970602 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.971311 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.971383 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:17.971417 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:17:17.971475 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260928 21:17:17.972082 12344 tablet_service.cc:1467] Processing CreateTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f (DEFAULT_TABLE table=test-table [id=56ea0d23df7046448d0ebec4a18305bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:17.972177 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.967749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:36429 (local address 127.0.0.1:45104)
0928 21:17:17.967901 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.967914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:17.967980 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:17.968692 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:17.968695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:17.968712 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:17.968799 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.968804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.969747 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.969750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.970962 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.970971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.971830 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.971833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.971917 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.971941 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:17:17.972013 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:17:17.972072 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":42}
I20260928 21:17:17.972256 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.968254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.5.18.195:36429)
0928 21:17:17.968458 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:17.968462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:17.968475 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:17.968535 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:17.968538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:17.968584 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:17.968667 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:17.968943 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.969657 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.971105 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.971724 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.971830 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.971893 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:17.972042 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:17:17.972126 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260928 21:17:17.972358 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955fb91b930a47d080c5cfd9db9bb51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:17.972447 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.967807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:46693 (local address 127.0.0.1:58374)
0928 21:17:17.968020 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.968033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:17.968113 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:17.969171 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:17.969174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:17.969190 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:17.969276 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.969282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.970113 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.970116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.971029 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:17.971036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.972112 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.972115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:17.972203 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.972224 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:17.972269 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:17:17.972334 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260928 21:17:17.972812 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.968469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.5.18.193:46693)
0928 21:17:17.968939 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:17.968943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:17.968956 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:17.969002 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:17.969005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:17.969061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:17.969146 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:17.969375 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.970003 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.971141 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:17.971990 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:17.972589 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:17.972637 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:17.972666 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:17:17.972719 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":59,"threads_started":1}
I20260928 21:17:17.973232 12207 tablet_service.cc:1467] Processing CreateTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f (DEFAULT_TABLE table=test-table [id=56ea0d23df7046448d0ebec4a18305bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:17.973431 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955fb91b930a47d080c5cfd9db9bb51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:17.974406 12641 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Bootstrap starting.
I20260928 21:17:17.974437 12481 tablet_service.cc:1467] Processing CreateTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f (DEFAULT_TABLE table=test-table [id=56ea0d23df7046448d0ebec4a18305bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:17.974680 12481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955fb91b930a47d080c5cfd9db9bb51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:17.975296 12641 tablet_bootstrap.cc:654] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:17.976457 12641 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: No bootstrap required, opened a new log
I20260928 21:17:17.976528 12641 ts_tablet_manager.cc:1397] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:17.976908 12644 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Bootstrap starting.
I20260928 21:17:17.976888 12641 raft_consensus.cc:348] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.976999 12641 raft_consensus.cc:374] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:17.977037 12641 raft_consensus.cc:729] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffb87801d7964812bf3162738fcccb5e, State: Initialized, Role: FOLLOWER
I20260928 21:17:17.977162 12641 consensus_queue.cc:260] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.977236 12642 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: Bootstrap starting.
I20260928 21:17:17.977442 12641 ts_tablet_manager.cc:1428] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:17.977842 12644 tablet_bootstrap.cc:654] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:17.978427 12642 tablet_bootstrap.cc:654] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:17.979353 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.978103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:34409)
0928 21:17:17.979205 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.979260 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":79,"threads_started":1}
I20260928 21:17:17.979489 12642 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: No bootstrap required, opened a new log
I20260928 21:17:17.979589 12642 ts_tablet_manager.cc:1397] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:17.979578 12644 tablet_bootstrap.cc:492] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: No bootstrap required, opened a new log
I20260928 21:17:17.979764 12644 ts_tablet_manager.cc:1397] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:17:17.979857 12410 heartbeater.cc:503] Master 127.5.18.252:44759 was elected leader, sending a full tablet report...
I20260928 21:17:17.979995 12642 raft_consensus.cc:348] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.980129 12642 raft_consensus.cc:374] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:17.980167 12642 raft_consensus.cc:729] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827fc91bd3704b08b1f5b9663c4d94d7, State: Initialized, Role: FOLLOWER
I20260928 21:17:17.980276 12642 consensus_queue.cc:260] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.980355 12644 raft_consensus.cc:348] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.980557 12644 raft_consensus.cc:374] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:17.980585 12642 ts_tablet_manager.cc:1428] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:17.980659 12644 raft_consensus.cc:729] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57163cb0239e46be85b86e8c04bb7dc9, State: Initialized, Role: FOLLOWER
I20260928 21:17:17.980868 12644 consensus_queue.cc:260] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:17.981262 12644 ts_tablet_manager.cc:1428] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:17.982167 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.981137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:44547)
0928 21:17:17.981437 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.982056 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260928 21:17:17.982834 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.981674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:51231)
0928 21:17:17.982690 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.982747 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":81,"threads_started":1}
I20260928 21:17:17.983078 12273 heartbeater.cc:503] Master 127.5.18.252:44759 was elected leader, sending a full tablet report...
I20260928 21:17:17.983376 12547 heartbeater.cc:503] Master 127.5.18.252:44759 was elected leader, sending a full tablet report...
I20260928 21:17:17.983592 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.983404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:59655)
0928 21:17:17.983489 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.983536 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 21:17:17.983634 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.983197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:39841)
0928 21:17:17.983541 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.983582 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260928 21:17:17.985371 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:17.979985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:50831)
0928 21:17:17.985253 (+ 5268us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:17.985301 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5205}
I20260928 21:17:18.148550 12650 raft_consensus.cc:482] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:18.148720 12650 raft_consensus.cc:504] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:18.149163 12650 leader_election.cc:288] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835), 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:18.149634 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955fb91b930a47d080c5cfd9db9bb51f" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" is_pre_election: true
I20260928 21:17:18.149643 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955fb91b930a47d080c5cfd9db9bb51f" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb87801d7964812bf3162738fcccb5e" is_pre_election: true
I20260928 21:17:18.149791 12227 raft_consensus.cc:2454] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 0.
I20260928 21:17:18.149791 12364 raft_consensus.cc:2454] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 0.
I20260928 21:17:18.150112 12434 leader_election.cc:302] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57163cb0239e46be85b86e8c04bb7dc9, 827fc91bd3704b08b1f5b9663c4d94d7, ffb87801d7964812bf3162738fcccb5e; no voters:
I20260928 21:17:18.150290 12650 raft_consensus.cc:2781] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:18.150357 12650 raft_consensus.cc:482] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:18.150386 12650 raft_consensus.cc:3037] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:18.151382 12650 raft_consensus.cc:504] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:18.151696 12650 leader_election.cc:288] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 election: Requested vote from peers ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835), 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693)
I20260928 21:17:18.152021 12364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955fb91b930a47d080c5cfd9db9bb51f" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffb87801d7964812bf3162738fcccb5e"
I20260928 21:17:18.152021 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955fb91b930a47d080c5cfd9db9bb51f" candidate_uuid: "57163cb0239e46be85b86e8c04bb7dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827fc91bd3704b08b1f5b9663c4d94d7"
I20260928 21:17:18.152143 12364 raft_consensus.cc:3037] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:18.152143 12227 raft_consensus.cc:3037] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:18.153177 12364 raft_consensus.cc:2454] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 1.
I20260928 21:17:18.153177 12227 raft_consensus.cc:2454] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57163cb0239e46be85b86e8c04bb7dc9 in term 1.
I20260928 21:17:18.153437 12434 leader_election.cc:302] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57163cb0239e46be85b86e8c04bb7dc9, 827fc91bd3704b08b1f5b9663c4d94d7, ffb87801d7964812bf3162738fcccb5e; no voters:
I20260928 21:17:18.153615 12650 raft_consensus.cc:2781] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:18.153823 12650 raft_consensus.cc:686] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 57163cb0239e46be85b86e8c04bb7dc9, State: Running, Role: LEADER
I20260928 21:17:18.154001 12650 consensus_queue.cc:237] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } }
I20260928 21:17:18.154474 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.154224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:50523)
0928 21:17:18.154340 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.154401 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:17:18.155625 12092 catalog_manager.cc:5495] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "57163cb0239e46be85b86e8c04bb7dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57163cb0239e46be85b86e8c04bb7dc9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36429 } health_report { overall_health: HEALTHY } } }
W20260928 21:17:18.161788 12550 tablet.cc:2369] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:18.209028 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.206249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.5.18.195:36429)
0928 21:17:18.206384 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:18.206387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:18.206398 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:18.206439 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:18.206441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:18.206484 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:18.206554 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:18.206785 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:18.207288 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:18.208092 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:18.208306 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:18.208392 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:18.208612 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:18.208683 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:18.208738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:18.208838 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:17:18.208894 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:17:18.211084 12364 raft_consensus.cc:1261] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Refusing update from remote peer 57163cb0239e46be85b86e8c04bb7dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:18.211084 12227 raft_consensus.cc:1261] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Refusing update from remote peer 57163cb0239e46be85b86e8c04bb7dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:18.211536 12650 consensus_queue.cc:1035] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffb87801d7964812bf3162738fcccb5e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:18.211706 12653 consensus_queue.cc:1035] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "827fc91bd3704b08b1f5b9663c4d94d7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 46693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:18.213356 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.213120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:39081)
0928 21:17:18.213215 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.213273 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:17:18.213464 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.213052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:37901)
0928 21:17:18.213337 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.213392 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260928 21:17:18.217201 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.216702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.195:56961)
0928 21:17:18.217066 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.217126 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260928 21:17:18.218065 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.217780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.193:53403)
0928 21:17:18.217947 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.217994 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 21:17:18.218756 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.218450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37051 (local address 127.5.18.194:53789)
0928 21:17:18.218643 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.218690 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 21:17:18.221419 12092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:18.221546 12092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:18.224756 12092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 693f0f79de6a4c5393aa1fd0f10f392e: Sending DeleteTablet for 3 replicas of tablet 955fb91b930a47d080c5cfd9db9bb51f
I20260928 21:17:18.225322 12344 tablet_service.cc:1514] Processing DeleteTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:18 UTC) from {username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:38280
I20260928 21:17:18.225500 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:18 UTC) from {username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:58374
I20260928 21:17:18.225673 12666 tablet_replica.cc:330] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: stopping tablet replica
I20260928 21:17:18.225786 12666 raft_consensus.cc:2229] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:18.225775 12481 tablet_service.cc:1514] Processing DeleteTablet for tablet 955fb91b930a47d080c5cfd9db9bb51f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:18 UTC) from {username='oryx', principal='oryx/127.5.18.252@KRBTEST.COM'} at 127.0.0.1:45104
I20260928 21:17:18.225947 12666 raft_consensus.cc:2258] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:18.226081 12668 tablet_replica.cc:330] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: stopping tablet replica
I20260928 21:17:18.225987 12667 tablet_replica.cc:330] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7: stopping tablet replica
I20260928 21:17:18.226194 12668 raft_consensus.cc:2229] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:18.226240 12667 raft_consensus.cc:2229] T 955fb91b930a47d080c5cfd9db9bb51f P 827fc91bd3704b08b1f5b9663c4d94d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:18.226405 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12056
I20260928 21:17:18.226413 12668 raft_consensus.cc:2258] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:18.227248 12666 ts_tablet_manager.cc:1905] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:18.228304 12668 ts_tablet_manager.cc:1905] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:18.229715 12666 ts_tablet_manager.cc:1918] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:18.229863 12666 log.cc:1196] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/955fb91b930a47d080c5cfd9db9bb51f
I20260928 21:17:18.230098 12666 ts_tablet_manager.cc:1939] T 955fb91b930a47d080c5cfd9db9bb51f P ffb87801d7964812bf3162738fcccb5e: Deleting consensus metadata
I20260928 21:17:18.230819 12668 ts_tablet_manager.cc:1918] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:18.230782 12079 catalog_manager.cc:4841] TS ffb87801d7964812bf3162738fcccb5e (127.5.18.194:43835): tablet 955fb91b930a47d080c5cfd9db9bb51f (table test-table [id=56ea0d23df7046448d0ebec4a18305bd]) successfully deleted
I20260928 21:17:18.230887 12668 log.cc:1196] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/955fb91b930a47d080c5cfd9db9bb51f
I20260928 21:17:18.231134 12668 ts_tablet_manager.cc:1939] T 955fb91b930a47d080c5cfd9db9bb51f P 57163cb0239e46be85b86e8c04bb7dc9: Deleting consensus metadata
I20260928 21:17:18.231691 12080 catalog_manager.cc:4841] TS 57163cb0239e46be85b86e8c04bb7dc9 (127.5.18.195:36429): tablet 955fb91b930a47d080c5cfd9db9bb51f (table test-table [id=56ea0d23df7046448d0ebec4a18305bd]) successfully deleted
I20260928 21:17:18.233083 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12281
I20260928 21:17:18.233263 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:18.232979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:46693 (local address 127.0.0.1:58380)
0928 21:17:18.233128 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:18.233186 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:46693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:17:18.235136 12078 catalog_manager.cc:4584] TS 827fc91bd3704b08b1f5b9663c4d94d7 (127.5.18.193:46693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 955fb91b930a47d080c5cfd9db9bb51f: Network error: Client connection negotiation failed: client connection to 127.5.18.193:46693: connect: Connection refused (error 111)
I20260928 21:17:18.238924 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12417
I20260928 21:17:18.244160 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11773
I20260928 21:17:18.249020 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 11882
2026-09-28T21:17:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5129 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:18 dist-test-slave-rc9f krb5kdc[12671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:18 dist-test-slave-rc9f krb5kdc[12671](info): set up 2 sockets
Sep 28 21:17:18 dist-test-slave-rc9f krb5kdc[12671](info): commencing operation
krb5kdc: starting...
W20260928 21:17:19.197914 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab.
Entry for principal oryx/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab.
I20260928 21:17:19.308226 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:40467
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab
--principal=oryx/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:19.412904 12684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:19.413147 12684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:19.413199 12684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:19.417068 12684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:19.417116 12684 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:17:19.417140 12684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:19.417160 12684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:19.417182 12684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:19.417209 12684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:19.422225 12684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:40467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.254.keytab
--principal=oryx/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:19.423290 12684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:19.424245 12684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:19.432027 12690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:19.432000 12689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:19.432106 12684 server_base.cc:1034] running on GCE node
W20260928 21:17:19.432016 12692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:19.432857 12684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:19.433853 12684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:19.435009 12684 hybrid_clock.cc:648] HybridClock initialized: now 1790630239434991 us; error 31 us; skew 500 ppm
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:19.437625 12684 init.cc:377] Logged in from keytab as oryx/127.5.18.254@KRBTEST.COM (short username oryx)
I20260928 21:17:19.438527 12684 webserver.cc:460] Webserver started at http://127.5.18.254:33685/ using document root <none> and password file <none>
I20260928 21:17:19.439040 12684 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:19.439105 12684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:19.439306 12684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:19.441236 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "b57eb581cb3349edb44282c9b1140f27"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "29c98df0046fb99176802bb3a320f3e6"
server_key_iv: "3b81c9ca46d66a1dd29d944c42bd848e"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.441828 12684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "b57eb581cb3349edb44282c9b1140f27"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "29c98df0046fb99176802bb3a320f3e6"
server_key_iv: "3b81c9ca46d66a1dd29d944c42bd848e"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.445338 12684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:17:19.447582 12699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:19.448799 12684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:19.448933 12684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "b57eb581cb3349edb44282c9b1140f27"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "29c98df0046fb99176802bb3a320f3e6"
server_key_iv: "3b81c9ca46d66a1dd29d944c42bd848e"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.449026 12684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:19.461004 12684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:19.463495 12684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:19.463650 12684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:19.470731 12684 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:40467
I20260928 21:17:19.470743 12761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:40467 every 8 connection(s)
I20260928 21:17:19.471599 12684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:19.472666 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12684
I20260928 21:17:19.472770 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:19.473018 5195 external_mini_cluster.cc:1428] Setting key 03e3a7da2e4593bb5caa0199890ad9cc
I20260928 21:17:19.474445 12762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:19.480383 12762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Bootstrap starting.
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:19.482779 12762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:19.483496 12762 log.cc:824] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:19.485435 12762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: No bootstrap required, opened a new log
I20260928 21:17:19.486814 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.474187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.5.18.254:40467)
0928 21:17:19.474557 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:19.474574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:19.474610 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:17:19.475212 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:19.475228 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:19.475541 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:19.475806 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:19.476673 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.477699 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.478784 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.479243 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.479356 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.482007 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:19.482012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:19.482024 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:19.482052 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:19.484285 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.484719 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.484723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.484727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.484824 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.485070 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.485072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.485073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.485416 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:19.485600 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:19.485864 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:17:19.486137 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260928 21:17:19.488379 12762 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } }
I20260928 21:17:19.488590 12762 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:19.488622 12762 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57eb581cb3349edb44282c9b1140f27, State: Initialized, Role: FOLLOWER
I20260928 21:17:19.488976 12762 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } }
I20260928 21:17:19.489100 12762 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:19.489158 12762 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:19.489243 12762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:19.490054 12762 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } }
I20260928 21:17:19.490346 12762 leader_election.cc:302] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57eb581cb3349edb44282c9b1140f27; no voters:
I20260928 21:17:19.490598 12762 leader_election.cc:288] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:19.490684 12767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:19.490872 12767 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [term 1 LEADER]: Becoming Leader. State: Replica: b57eb581cb3349edb44282c9b1140f27, State: Running, Role: LEADER
I20260928 21:17:19.491583 12767 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } }
I20260928 21:17:19.491791 12762 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:19.493124 12768 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57eb581cb3349edb44282c9b1140f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } } }
I20260928 21:17:19.493165 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57eb581cb3349edb44282c9b1140f27. Latest consensus state: current_term: 1 leader_uuid: "b57eb581cb3349edb44282c9b1140f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57eb581cb3349edb44282c9b1140f27" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 40467 } } }
I20260928 21:17:19.493229 12768 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:19.493264 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:19.494048 12779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:19.496821 12779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:19.501850 12779 catalog_manager.cc:1347] Generated new cluster ID: 954084e4b6dc4e4ab3c42fa27011498e
I20260928 21:17:19.501911 12779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:19.532717 12779 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:19.533984 12779 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:19.544936 12779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Generated new TSK 0
I20260928 21:17:19.545794 12779 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
I20260928 21:17:19.575624 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:40467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:19.684442 12788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:19.684736 12788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:19.684782 12788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:19.688933 12788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:19.689020 12788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:19.689060 12788 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:17:19.689100 12788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:19.694752 12788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:19.695853 12788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:19.696875 12788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:19.706162 12793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:19.706175 12794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:19.706159 12796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:19.706408 12788 server_base.cc:1034] running on GCE node
I20260928 21:17:19.706820 12788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:19.707427 12788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:19.708602 12788 hybrid_clock.cc:648] HybridClock initialized: now 1790630239708586 us; error 37 us; skew 500 ppm
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:19.711232 12788 init.cc:377] Logged in from keytab as oryx/127.5.18.193@KRBTEST.COM (short username oryx)
I20260928 21:17:19.712039 12788 webserver.cc:460] Webserver started at http://127.5.18.193:46699/ using document root <none> and password file <none>
I20260928 21:17:19.712543 12788 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:19.712599 12788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:19.712751 12788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:19.714725 12788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "431de377836a44b99c60264008f9d483"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "e539ce9314f19e5b20fd0aa86d0bf375"
server_key_iv: "bfe4a0902e10e8702f959ee6efd9e8b0"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.715237 12788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "431de377836a44b99c60264008f9d483"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "e539ce9314f19e5b20fd0aa86d0bf375"
server_key_iv: "bfe4a0902e10e8702f959ee6efd9e8b0"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.718760 12788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:19.721146 12803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:19.722332 12788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:19.722433 12788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "431de377836a44b99c60264008f9d483"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "e539ce9314f19e5b20fd0aa86d0bf375"
server_key_iv: "bfe4a0902e10e8702f959ee6efd9e8b0"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.722538 12788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:19.743883 12788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:19.746819 12788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:19.746989 12788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:19.748382 12788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:19.748430 12788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:19.748490 12788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:19.748517 12788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:19.758595 12788 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45709
I20260928 21:17:19.758634 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45709 every 8 connection(s)
I20260928 21:17:19.759544 12788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:19.760738 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.749110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:55281 (local address 127.5.18.254:40467)
0928 21:17:19.749219 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:19.749223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:19.750278 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 21:17:19.751234 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:19.751238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:19.751291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:19.751341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:19.752804 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.753420 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.754547 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.754783 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.754872 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.757257 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:19.757261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:19.757265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:19.757296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:19.758772 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.759215 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.759218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.759220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.759307 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.759591 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.759594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.759596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.759811 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:19.759911 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:19.760445 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 21:17:19.760579 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:19.761168 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12788
I20260928 21:17:19.761282 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:19.761493 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.749668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.193:55281)
0928 21:17:19.750135 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:19.750177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:19.751014 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:19.751729 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:19.751736 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:19.752047 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:19.752626 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.752647 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.753532 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.753536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.754426 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.754435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.754928 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.754932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.755120 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.755665 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:19.755681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:19.757100 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:19.758866 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.758871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.758882 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.759120 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.759385 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.759387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.759389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.759507 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.759915 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:19.759921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:19.760205 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 21:17:19.760393 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 21:17:19.760692 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260928 21:17:19.761543 5195 external_mini_cluster.cc:1428] Setting key cf13e4b93edbb4710ad720824721d95f
I20260928 21:17:19.762650 12919 heartbeater.cc:347] Connected to a master server at 127.5.18.254:40467
I20260928 21:17:19.762784 12919 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:19.763269 12919 heartbeater.cc:511] Master 127.5.18.254:40467 requested a full tablet report, sending...
I20260928 21:17:19.764631 12715 ts_manager.cc:194] Registered new tserver with Master: 431de377836a44b99c60264008f9d483 (127.5.18.193:45709)
I20260928 21:17:19.766131 12715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:55281
WARNING: no policy specified for oryx/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:19.775712 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.764801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:55729 (local address 127.5.18.254:40467)
0928 21:17:19.764906 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:19.764909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:19.769806 (+ 4897us) server_negotiation.cc:407] Connection header received
0928 21:17:19.769882 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:19.769884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:19.769921 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:19.769987 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:19.770496 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.770966 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.771846 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.772055 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.772140 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.773072 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:19.773075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:19.773077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:19.773096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:19.774364 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.774719 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.774722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.774724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.774800 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.775066 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.775068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.775069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.775214 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:19.775284 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:19.775522 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:17:19.775604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:17:19.775784 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.764729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.193:55729)
0928 21:17:19.769729 (+ 5000us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:19.769742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:19.769825 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:19.770036 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:19.770039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:19.770259 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:19.770392 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.770400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.771091 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.771095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.771755 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.771764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.772155 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.772159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.772248 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.772648 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:19.772658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:19.772941 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:19.774459 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.774461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.774463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.774642 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.774881 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.774884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.774886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.774976 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.775285 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:19.775288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:19.775377 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 21:17:19.775473 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:17:19.775598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4925,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
I20260928 21:17:19.787849 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:40467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:19.891268 12928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:19.891538 12928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:19.891601 12928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:19.895644 12928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:19.895700 12928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:19.895738 12928 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:17:19.895767 12928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:19.900722 12928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:19.901676 12928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:19.902738 12928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:19.911656 12934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:19.911715 12928 server_base.cc:1034] running on GCE node
W20260928 21:17:19.911649 12933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:19.911649 12936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:19.912333 12928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:19.912917 12928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:19.914057 12928 hybrid_clock.cc:648] HybridClock initialized: now 1790630239914041 us; error 28 us; skew 500 ppm
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:19.916576 12928 init.cc:377] Logged in from keytab as oryx/127.5.18.194@KRBTEST.COM (short username oryx)
I20260928 21:17:19.917359 12928 webserver.cc:460] Webserver started at http://127.5.18.194:44229/ using document root <none> and password file <none>
I20260928 21:17:19.917887 12928 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:19.917944 12928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:19.918098 12928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:19.919967 12928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "c94d26053e0647d881d0118cc12d0153"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "b3e7f0df06da22727274fa49526dc9d9"
server_key_iv: "a43abc437c3247ab771a4f2fa1df921b"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.920445 12928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "c94d26053e0647d881d0118cc12d0153"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "b3e7f0df06da22727274fa49526dc9d9"
server_key_iv: "a43abc437c3247ab771a4f2fa1df921b"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.923856 12928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:19.926100 12943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:19.927261 12928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:19.927352 12928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c94d26053e0647d881d0118cc12d0153"
format_stamp: "Formatted at 2026-09-28 21:17:19 on dist-test-slave-rc9f"
server_key: "b3e7f0df06da22727274fa49526dc9d9"
server_key_iv: "a43abc437c3247ab771a4f2fa1df921b"
server_key_version: "encryptionkey@0"
I20260928 21:17:19.927452 12928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:19.937294 12928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:19.940116 12928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:19.940270 12928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:19.941635 12928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:19.941682 12928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:19.941723 12928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:19.941749 12928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:17:19 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:19.951509 12928 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:38643
I20260928 21:17:19.951514 13058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:38643 every 8 connection(s)
I20260928 21:17:19.952561 12928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:19.953143 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 12928
I20260928 21:17:19.953284 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:19.953706 5195 external_mini_cluster.cc:1428] Setting key 99cddaf52cf00858585ed0637847e3f3
I20260928 21:17:19.954896 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.942373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:51889 (local address 127.5.18.254:40467)
0928 21:17:19.942508 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:19.942513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:19.943395 (+ 882us) server_negotiation.cc:407] Connection header received
0928 21:17:19.944557 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:19.944559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:19.944615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:19.944708 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:19.946171 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.946777 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.947916 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.948265 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.948372 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.950794 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:19.950798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:19.950801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:19.950828 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:19.952392 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.953061 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.953065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.953067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.953225 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.953729 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.953731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.953733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.953932 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:19.954017 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:19.954603 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 21:17:19.954727 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:17:19.955632 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.942737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.194:51889)
0928 21:17:19.943231 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:19.943287 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:19.944334 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:19.944903 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:19.944914 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:19.945349 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:19.946033 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.946052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.946890 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.946893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.947791 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.947801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.948414 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.948419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.948609 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.949164 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:19.949179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:19.950641 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:19.952500 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.952504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.952521 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.952920 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.953411 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.953414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.953416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.953636 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.954033 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:19.954039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:19.954329 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 21:17:19.954528 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 21:17:19.954842 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260928 21:17:19.956812 13059 heartbeater.cc:347] Connected to a master server at 127.5.18.254:40467
I20260928 21:17:19.956945 13059 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:19.957388 13059 heartbeater.cc:511] Master 127.5.18.254:40467 requested a full tablet report, sending...
I20260928 21:17:19.958328 12715 ts_manager.cc:194] Registered new tserver with Master: c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:19.958892 12715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:51889
WARNING: no policy specified for oryx/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:19.966383 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.959109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:51987 (local address 127.5.18.254:40467)
0928 21:17:19.959216 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:19.959221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:19.959630 (+ 409us) server_negotiation.cc:407] Connection header received
0928 21:17:19.959748 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:19.959751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:19.959801 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:19.959884 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:19.960344 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.961098 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.961915 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.962256 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.962395 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.963444 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:19.963448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:19.963451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:19.963478 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:19.964722 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.965136 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.965140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.965142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.965211 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:19.965487 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:19.965491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:19.965493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:19.965795 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:19.965903 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:19.966166 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 21:17:19.966259 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:17:19.966528 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:19.959259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.194:51987)
0928 21:17:19.959539 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:19.959553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:19.959653 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:19.959908 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:19.959911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:19.960112 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:19.960221 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.960228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.961221 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.961223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.961808 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:19.961817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:19.962390 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:19.962393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:19.962464 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:19.962977 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:19.962987 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:19.963322 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:19.964817 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.964820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.964822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.965052 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.965289 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:19.965292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:19.965293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:19.965398 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:19.965899 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:19.965902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:19.965993 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 21:17:19.966112 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:17:19.966241 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
I20260928 21:17:19.979784 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:40467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:20.083757 13068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:20.084002 13068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:20.084056 13068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:20.087939 13068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:20.087996 13068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:20.088044 13068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:17:20.088093 13068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:20.093263 13068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.18.254:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:20.094318 13068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:20.095285 13068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:20.104208 13073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:20.104274 13076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:20.104419 13068 server_base.cc:1034] running on GCE node
W20260928 21:17:20.104208 13074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:20.104832 13068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:20.105381 13068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:20.106534 13068 hybrid_clock.cc:648] HybridClock initialized: now 1790630240106505 us; error 40 us; skew 500 ppm
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:20.109089 13068 init.cc:377] Logged in from keytab as oryx/127.5.18.195@KRBTEST.COM (short username oryx)
I20260928 21:17:20.109930 13068 webserver.cc:460] Webserver started at http://127.5.18.195:40953/ using document root <none> and password file <none>
I20260928 21:17:20.110417 13068 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:20.110477 13068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:20.110635 13068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:20.112568 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "dd69b0a60023499689e1c3cfccaa9554"
format_stamp: "Formatted at 2026-09-28 21:17:20 on dist-test-slave-rc9f"
server_key: "de614446869b7e4d0363e8582809e5bd"
server_key_iv: "4e6573cec08103476d3bf8e9b7941203"
server_key_version: "encryptionkey@0"
I20260928 21:17:20.113058 13068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "dd69b0a60023499689e1c3cfccaa9554"
format_stamp: "Formatted at 2026-09-28 21:17:20 on dist-test-slave-rc9f"
server_key: "de614446869b7e4d0363e8582809e5bd"
server_key_iv: "4e6573cec08103476d3bf8e9b7941203"
server_key_version: "encryptionkey@0"
I20260928 21:17:20.116405 13068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:20.118603 13083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:20.119737 13068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.119824 13068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "dd69b0a60023499689e1c3cfccaa9554"
format_stamp: "Formatted at 2026-09-28 21:17:20 on dist-test-slave-rc9f"
server_key: "de614446869b7e4d0363e8582809e5bd"
server_key_iv: "4e6573cec08103476d3bf8e9b7941203"
server_key_version: "encryptionkey@0"
I20260928 21:17:20.119910 13068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:20.139575 13068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:20.142196 13068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:20.142340 13068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:20.143687 13068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:20.143738 13068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:20.143791 13068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:20.143818 13068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:20.153162 13068 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:38115
I20260928 21:17:20.153221 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:38115 every 8 connection(s)
I20260928 21:17:20.154039 13068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:20.155122 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13068
I20260928 21:17:20.155237 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:20.155498 5195 external_mini_cluster.cc:1428] Setting key f44b6e6cacb154672949c2720223cf97
I20260928 21:17:20.156697 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.144320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:48165 (local address 127.5.18.254:40467)
0928 21:17:20.144458 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.144463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.145397 (+ 934us) server_negotiation.cc:407] Connection header received
0928 21:17:20.146466 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.146470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.146529 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.146587 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.148176 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.148804 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.150246 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.150460 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.150555 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.153084 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:20.153087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:20.153089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:20.153116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:20.154581 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.155142 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.155146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.155148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.155245 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.155558 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.155561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.155563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.155783 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:20.155868 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.156439 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.156549 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:17:20.157655 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.144795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.195:48165)
0928 21:17:20.145240 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.145287 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.146223 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.146995 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.147015 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.147383 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:20.148037 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.148058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.148920 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.148923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.150134 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.150144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.150597 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.150600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.150776 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.151327 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:20.151346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:20.152891 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:20.154704 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.154710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.154724 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.155040 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.155343 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.155345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.155346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.155471 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.155894 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:20.155899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:20.156203 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.156395 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.156682 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260928 21:17:20.158809 13199 heartbeater.cc:347] Connected to a master server at 127.5.18.254:40467
I20260928 21:17:20.158941 13199 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:20.159416 13199 heartbeater.cc:511] Master 127.5.18.254:40467 requested a full tablet report, sending...
I20260928 21:17:20.160363 12714 ts_manager.cc:194] Registered new tserver with Master: dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115)
I20260928 21:17:20.161023 12714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:48165
I20260928 21:17:20.166641 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.160714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:36625 (local address 127.5.18.254:40467)
0928 21:17:20.160816 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.160820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.160978 (+ 158us) server_negotiation.cc:407] Connection header received
0928 21:17:20.161075 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.161078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.161121 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.161183 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.161633 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.162160 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.162960 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.163174 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.163256 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.164230 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:20.164234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:20.164237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:20.164260 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:20.165248 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.165624 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.165628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.165630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.165702 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.165932 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.165935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.165937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.166100 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:20.166181 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.166405 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.166500 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:17:20.166656 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.160667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.5.18.195:36625)
0928 21:17:20.160909 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.160920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.161005 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.161196 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.161199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.161397 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:20.161523 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.161531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.162260 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.162264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.162856 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.162864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.163311 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.163314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.163403 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.163826 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:20.163836 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:20.164115 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:20.165339 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.165342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.165345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.165528 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.165778 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.165782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.165783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.165858 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.166182 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:20.166184 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:20.166267 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.166348 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.166476 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
I20260928 21:17:20.169189 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:17:20.189976 12765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:20.184554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.5.18.254:40467)
0928 21:17:20.184665 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.184669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.184681 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:20.184717 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.184720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.184756 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.184821 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.185459 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.186008 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.186794 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.186990 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.187062 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.189866 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:20.198082 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.191242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.5.18.254:40467)
0928 21:17:20.191352 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.191355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.191463 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:17:20.191511 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.191513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.191553 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.191642 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.192168 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.192663 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.193471 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.193710 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.193784 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.195659 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:20.195662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:20.195664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:20.195684 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:20.196926 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.197221 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.197223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.197224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.197290 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.197454 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.197456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.197457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.197647 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:20.197736 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.197897 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.197976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:17:20.200562 12714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:20.203473 12714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:20.216717 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.211032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.5.18.193:45709)
0928 21:17:20.211959 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.211971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.212007 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:17:20.212078 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.212083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.212283 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.212721 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.212841 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.214015 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.215492 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216102 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216193 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216357 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.216484 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.216577 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":115,"threads_started":1}
I20260928 21:17:20.216707 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.210870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45709 (local address 127.0.0.1:59338)
0928 21:17:20.211518 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.211578 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.211693 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.212447 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.212453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.212474 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.212603 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.212609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.214132 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.214136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.215360 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.215369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216205 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.216305 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216350 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.216450 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.216562 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260928 21:17:20.216924 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.211054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38643 (local address 127.0.0.1:46088)
0928 21:17:20.211709 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.211720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.211782 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.212851 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.212853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.212869 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.212954 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.212960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.213863 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.213866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.215403 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.215410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216377 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.216514 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216561 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.216618 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.216716 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":124,"thread_start_us":73,"threads_started":1}
I20260928 21:17:20.216928 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.211647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.5.18.195:38115)
0928 21:17:20.211980 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.211991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.212009 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:17:20.212078 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.212083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.212283 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.212430 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.212817 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.214015 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.215335 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216104 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216205 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216356 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.216702 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.216791 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260928 21:17:20.217170 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.212041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.5.18.194:38643)
0928 21:17:20.212409 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.212421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.212441 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:17:20.212510 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.212515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.212709 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.212881 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.213039 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.213769 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.215513 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216255 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216766 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216910 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.216956 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.217039 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":82,"threads_started":1}
I20260928 21:17:20.217478 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.211122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:38115 (local address 127.0.0.1:49016)
0928 21:17:20.211774 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.211787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.211872 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.212546 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.212549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.212565 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.212701 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.212707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.214154 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.214157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.215230 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.215240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.216402 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.216405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.216500 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.216557 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.216620 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.217353 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":758,"thread_start_us":76,"threads_started":1}
I20260928 21:17:20.219771 12853 tablet_service.cc:1467] Processing CreateTablet for tablet 80460e4fafb647078f7fecabb56f0429 (DEFAULT_TABLE table=test-table [id=513c3fe866ac4228b7cfd934562f880d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.219888 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 80460e4fafb647078f7fecabb56f0429 (DEFAULT_TABLE table=test-table [id=513c3fe866ac4228b7cfd934562f880d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.219888 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 80460e4fafb647078f7fecabb56f0429 (DEFAULT_TABLE table=test-table [id=513c3fe866ac4228b7cfd934562f880d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.220667 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80460e4fafb647078f7fecabb56f0429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.220837 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80460e4fafb647078f7fecabb56f0429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.220837 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80460e4fafb647078f7fecabb56f0429. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.227756 13227 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Bootstrap starting.
I20260928 21:17:20.228757 13229 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Bootstrap starting.
I20260928 21:17:20.229744 13228 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Bootstrap starting.
I20260928 21:17:20.230019 13227 tablet_bootstrap.cc:654] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.230831 13229 tablet_bootstrap.cc:654] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.230865 13227 log.cc:824] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:20.231530 13228 tablet_bootstrap.cc:654] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.231546 13229 log.cc:824] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:20.232288 13228 log.cc:824] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:20.232918 13227 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: No bootstrap required, opened a new log
I20260928 21:17:20.233100 13227 ts_tablet_manager.cc:1397] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:17:20.233728 13229 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: No bootstrap required, opened a new log
I20260928 21:17:20.233896 13229 ts_tablet_manager.cc:1397] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:17:20.233929 13228 tablet_bootstrap.cc:492] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: No bootstrap required, opened a new log
I20260928 21:17:20.234092 13228 ts_tablet_manager.cc:1397] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 21:17:20.236745 13228 raft_consensus.cc:348] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.236951 13228 raft_consensus.cc:374] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.237001 13228 raft_consensus.cc:729] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.236827 13227 raft_consensus.cc:348] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.237062 13227 raft_consensus.cc:374] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.237106 13227 raft_consensus.cc:729] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431de377836a44b99c60264008f9d483, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.237444 13228 consensus_queue.cc:260] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.237483 13227 consensus_queue.cc:260] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.237406 13229 raft_consensus.cc:348] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.237632 13229 raft_consensus.cc:374] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.237673 13229 raft_consensus.cc:729] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.238123 12919 heartbeater.cc:503] Master 127.5.18.254:40467 was elected leader, sending a full tablet report...
I20260928 21:17:20.238070 13229 consensus_queue.cc:260] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.238423 13228 ts_tablet_manager.cc:1428] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 21:17:20.238425 13227 ts_tablet_manager.cc:1428] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:17:20.238729 13199 heartbeater.cc:503] Master 127.5.18.254:40467 was elected leader, sending a full tablet report...
I20260928 21:17:20.239684 13229 ts_tablet_manager.cc:1428] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:17:20.239979 13059 heartbeater.cc:503] Master 127.5.18.254:40467 was elected leader, sending a full tablet report...
W20260928 21:17:20.260720 12920 tablet.cc:2369] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:20.388438 13235 raft_consensus.cc:482] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:20.388649 13235 raft_consensus.cc:504] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.389820 13235 leader_election.cc:288] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115)
I20260928 21:17:20.394281 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.390120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:38115 (local address 127.5.18.194:41053)
0928 21:17:20.390228 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.390244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.390348 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.390639 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.390642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.390662 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.390771 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.390777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.391805 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.391809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.393022 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.393031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.393886 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.393889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.393979 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.394007 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.394081 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.394154 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 21:17:20.394289 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.390321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:41053 (local address 127.5.18.195:38115)
0928 21:17:20.390434 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.390437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.390449 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:20.390494 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.390497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.390538 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.390626 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.390896 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.391703 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.393128 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.393783 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.393900 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.393955 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.394099 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.394153 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:20.394987 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.390162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:50673 (local address 127.5.18.193:45709)
0928 21:17:20.390281 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.390285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.390800 (+ 515us) server_negotiation.cc:407] Connection header received
0928 21:17:20.390906 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.390908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.390949 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.391007 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.391829 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.392625 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.393989 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.394520 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.394613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.394647 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.394827 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.394879 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:20.395005 13236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.390123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45709 (local address 127.5.18.194:50673)
0928 21:17:20.390714 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.390726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.390807 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.391024 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.391027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.391043 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.391718 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.391727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.392713 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.392716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.393889 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.393898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.394645 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.394649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.394725 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.394742 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.394809 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.394871 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":295,"thread_start_us":126,"threads_started":1}
I20260928 21:17:20.395236 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80460e4fafb647078f7fecabb56f0429" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd69b0a60023499689e1c3cfccaa9554" is_pre_election: true
I20260928 21:17:20.395522 13153 raft_consensus.cc:2454] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 0.
I20260928 21:17:20.395581 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80460e4fafb647078f7fecabb56f0429" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483" is_pre_election: true
I20260928 21:17:20.395859 12873 raft_consensus.cc:2454] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 0.
I20260928 21:17:20.395987 12945 leader_election.cc:302] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:20.396243 13235 raft_consensus.cc:2781] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:20.396324 13235 raft_consensus.cc:482] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:20.396371 13235 raft_consensus.cc:3037] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.397508 13235 raft_consensus.cc:504] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.397854 13235 leader_election.cc:288] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 election: Requested vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115)
I20260928 21:17:20.398167 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80460e4fafb647078f7fecabb56f0429" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd69b0a60023499689e1c3cfccaa9554"
I20260928 21:17:20.398206 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80460e4fafb647078f7fecabb56f0429" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483"
I20260928 21:17:20.398278 13153 raft_consensus.cc:3037] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.398313 12873 raft_consensus.cc:3037] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.399281 13153 raft_consensus.cc:2454] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 1.
I20260928 21:17:20.399286 12873 raft_consensus.cc:2454] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 1.
I20260928 21:17:20.399550 12945 leader_election.cc:302] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 431de377836a44b99c60264008f9d483, c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:20.399713 13235 raft_consensus.cc:2781] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:20.399904 13235 raft_consensus.cc:686] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 1 LEADER]: Becoming Leader. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Running, Role: LEADER
I20260928 21:17:20.400431 13235 consensus_queue.cc:237] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.403312 12714 catalog_manager.cc:5495] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94d26053e0647d881d0118cc12d0153 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "c94d26053e0647d881d0118cc12d0153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } health_report { overall_health: UNKNOWN } } }
W20260928 21:17:20.405154 13200 tablet.cc:2369] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:20.450708 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.447092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.5.18.194:38643)
0928 21:17:20.447197 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.447201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.447213 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:20.447410 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.447412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.447475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.447571 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:20.447804 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.448358 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.449232 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.449459 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.449552 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.449905 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.450370 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.450465 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.450533 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.450591 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 21:17:20.453999 13060 tablet.cc:2369] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:20.456388 13153 raft_consensus.cc:1261] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Refusing update from remote peer c94d26053e0647d881d0118cc12d0153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.456413 12873 raft_consensus.cc:1261] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Refusing update from remote peer c94d26053e0647d881d0118cc12d0153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.457041 13237 consensus_queue.cc:1035] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.457219 13235 consensus_queue.cc:1035] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.465754 13242 mvcc.cc:205] Tried to move back new op lower bound from 7334421464903159808 to 7334421464682545152. Current Snapshot: MvccSnapshot[applied={T|T < 7334421464903159808}]
I20260928 21:17:20.474499 12714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:20.474681 12714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:20.477355 12714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Sending DeleteTablet for 3 replicas of tablet 80460e4fafb647078f7fecabb56f0429
I20260928 21:17:20.478186 12993 tablet_service.cc:1514] Processing DeleteTablet for tablet 80460e4fafb647078f7fecabb56f0429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:46088
I20260928 21:17:20.478408 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet 80460e4fafb647078f7fecabb56f0429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49016
I20260928 21:17:20.478500 12853 tablet_service.cc:1514] Processing DeleteTablet for tablet 80460e4fafb647078f7fecabb56f0429 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 21:17:20.478895 13250 tablet_replica.cc:330] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: stopping tablet replica
I20260928 21:17:20.479223 13250 raft_consensus.cc:2229] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:20.479565 13250 raft_consensus.cc:2258] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.479516 12714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:20.479959 12714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:20.481263 13250 ts_tablet_manager.cc:1905] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.481534 13251 tablet_replica.cc:330] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: stopping tablet replica
I20260928 21:17:20.481973 13251 raft_consensus.cc:2229] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:20.484601 13250 ts_tablet_manager.cc:1918] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:20.484972 13251 raft_consensus.cc:2258] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.485040 12853 tablet_service.cc:1467] Processing CreateTablet for tablet a235388fe78047299574ff1415e555a7 (DEFAULT_TABLE table=test-table [id=055c32c7799045fa9916fa261077719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.485293 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a235388fe78047299574ff1415e555a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.485387 13250 log.cc:1196] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/80460e4fafb647078f7fecabb56f0429
I20260928 21:17:20.485390 12993 tablet_service.cc:1467] Processing CreateTablet for tablet a235388fe78047299574ff1415e555a7 (DEFAULT_TABLE table=test-table [id=055c32c7799045fa9916fa261077719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.485700 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a235388fe78047299574ff1415e555a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.486166 13250 ts_tablet_manager.cc:1939] T 80460e4fafb647078f7fecabb56f0429 P dd69b0a60023499689e1c3cfccaa9554: Deleting consensus metadata
I20260928 21:17:20.487789 13227 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Bootstrap starting.
I20260928 21:17:20.488171 13229 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Bootstrap starting.
I20260928 21:17:20.488657 13227 tablet_bootstrap.cc:654] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.488862 13251 ts_tablet_manager.cc:1905] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.489149 13229 tablet_bootstrap.cc:654] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.489982 13227 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: No bootstrap required, opened a new log
I20260928 21:17:20.490082 13227 ts_tablet_manager.cc:1397] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.490226 12701 catalog_manager.cc:4841] TS dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115): tablet 80460e4fafb647078f7fecabb56f0429 (table test-table [id=513c3fe866ac4228b7cfd934562f880d]) successfully deleted
I20260928 21:17:20.490533 13227 raft_consensus.cc:348] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.490649 13227 raft_consensus.cc:374] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.490685 13227 raft_consensus.cc:729] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431de377836a44b99c60264008f9d483, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.490649 13133 tablet_service.cc:1467] Processing CreateTablet for tablet a235388fe78047299574ff1415e555a7 (DEFAULT_TABLE table=test-table [id=055c32c7799045fa9916fa261077719f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.490796 13227 consensus_queue.cc:260] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.490909 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a235388fe78047299574ff1415e555a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.491058 13227 ts_tablet_manager.cc:1428] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.492029 13251 ts_tablet_manager.cc:1918] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:20.492159 13251 log.cc:1196] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/80460e4fafb647078f7fecabb56f0429
I20260928 21:17:20.492511 13251 ts_tablet_manager.cc:1939] T 80460e4fafb647078f7fecabb56f0429 P 431de377836a44b99c60264008f9d483: Deleting consensus metadata
I20260928 21:17:20.493185 13228 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Bootstrap starting.
I20260928 21:17:20.493794 12700 catalog_manager.cc:4841] TS 431de377836a44b99c60264008f9d483 (127.5.18.193:45709): tablet 80460e4fafb647078f7fecabb56f0429 (table test-table [id=513c3fe866ac4228b7cfd934562f880d]) successfully deleted
I20260928 21:17:20.494117 13228 tablet_bootstrap.cc:654] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.494545 13249 tablet_replica.cc:330] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: stopping tablet replica
I20260928 21:17:20.494810 13249 raft_consensus.cc:2229] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:20.494920 13229 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: No bootstrap required, opened a new log
I20260928 21:17:20.495011 13229 ts_tablet_manager.cc:1397] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 21:17:20.495139 13249 raft_consensus.cc:2258] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.495473 13229 raft_consensus.cc:348] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.495599 13229 raft_consensus.cc:374] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.495633 13229 raft_consensus.cc:729] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.495749 13229 consensus_queue.cc:260] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.495992 13228 tablet_bootstrap.cc:492] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: No bootstrap required, opened a new log
I20260928 21:17:20.496022 13229 ts_tablet_manager.cc:1428] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.496061 13228 ts_tablet_manager.cc:1397] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 21:17:20.496443 13249 ts_tablet_manager.cc:1905] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.496444 13228 raft_consensus.cc:348] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.496551 13228 raft_consensus.cc:374] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.496579 13228 raft_consensus.cc:729] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.496692 13228 consensus_queue.cc:260] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.496924 13228 ts_tablet_manager.cc:1428] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:17:20.498948 13249 ts_tablet_manager.cc:1918] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:20.499049 13249 log.cc:1196] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/80460e4fafb647078f7fecabb56f0429
I20260928 21:17:20.499344 13249 ts_tablet_manager.cc:1939] T 80460e4fafb647078f7fecabb56f0429 P c94d26053e0647d881d0118cc12d0153: Deleting consensus metadata
I20260928 21:17:20.500279 12702 catalog_manager.cc:4841] TS c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643): tablet 80460e4fafb647078f7fecabb56f0429 (table test-table [id=513c3fe866ac4228b7cfd934562f880d]) successfully deleted
I20260928 21:17:20.514551 13235 raft_consensus.cc:482] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:20.514653 13235 raft_consensus.cc:504] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.515008 13235 leader_election.cc:288] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115)
I20260928 21:17:20.515365 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a235388fe78047299574ff1415e555a7" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd69b0a60023499689e1c3cfccaa9554" is_pre_election: true
I20260928 21:17:20.515393 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a235388fe78047299574ff1415e555a7" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483" is_pre_election: true
I20260928 21:17:20.515506 13153 raft_consensus.cc:2454] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 0.
I20260928 21:17:20.515519 12873 raft_consensus.cc:2454] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 0.
I20260928 21:17:20.515802 12945 leader_election.cc:302] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:20.515954 13235 raft_consensus.cc:2781] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:20.516000 13235 raft_consensus.cc:482] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:20.516028 13235 raft_consensus.cc:3037] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.516933 13235 raft_consensus.cc:504] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.517238 13235 leader_election.cc:288] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 election: Requested vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115)
I20260928 21:17:20.517530 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a235388fe78047299574ff1415e555a7" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483"
I20260928 21:17:20.517652 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a235388fe78047299574ff1415e555a7" candidate_uuid: "c94d26053e0647d881d0118cc12d0153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd69b0a60023499689e1c3cfccaa9554"
I20260928 21:17:20.517704 12873 raft_consensus.cc:3037] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.517791 13153 raft_consensus.cc:3037] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.518595 12873 raft_consensus.cc:2454] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 1.
I20260928 21:17:20.518843 12944 leader_election.cc:302] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431de377836a44b99c60264008f9d483, c94d26053e0647d881d0118cc12d0153; no voters:
I20260928 21:17:20.518939 13153 raft_consensus.cc:2454] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94d26053e0647d881d0118cc12d0153 in term 1.
I20260928 21:17:20.518990 13235 raft_consensus.cc:2781] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:20.519068 13235 raft_consensus.cc:686] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 1 LEADER]: Becoming Leader. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Running, Role: LEADER
I20260928 21:17:20.519192 13235 consensus_queue.cc:237] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.520694 12713 catalog_manager.cc:5495] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94d26053e0647d881d0118cc12d0153 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "c94d26053e0647d881d0118cc12d0153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630239, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.254@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:20.552126 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.543615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:40467 (local address 127.0.0.1:47588)
0928 21:17:20.543898 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.543912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.544025 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.544248 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.544251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.544762 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:20.544884 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.544891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.545594 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.545598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.546497 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.546508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.546894 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.546898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.546986 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.547527 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:20.547541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:20.549089 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:20.550571 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.550580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.550587 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.550872 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.551091 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:20.551094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:20.551095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:20.551185 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:20.551484 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:20.551490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:20.551627 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.551757 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.551928 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260928 21:17:20.552155 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.543695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.5.18.254:40467)
0928 21:17:20.543825 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.543830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.543982 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:17:20.544105 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.544107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.544144 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.544207 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.544980 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.545476 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.546598 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.546804 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.546872 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.549214 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:20.549217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:20.549219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:20.549240 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:20.550465 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.550954 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.550957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.550958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.551024 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.551252 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.551254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.551255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.551406 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:20.551470 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.551814 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.551994 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":52}
I20260928 21:17:20.555900 12713 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:47588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:17:20.556632 12713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:20.561143 12853 tablet_service.cc:1467] Processing CreateTablet for tablet 61a8793760114adebaf276ee1348989c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7f898d380394110866e7eb059afc3a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:17:20.561913 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 61a8793760114adebaf276ee1348989c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7f898d380394110866e7eb059afc3a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:17:20.562260 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a8793760114adebaf276ee1348989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.564934 13229 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: Bootstrap starting.
I20260928 21:17:20.565281 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 61a8793760114adebaf276ee1348989c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c7f898d380394110866e7eb059afc3a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:17:20.565668 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a8793760114adebaf276ee1348989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.565773 13229 tablet_bootstrap.cc:654] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.566992 13229 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: No bootstrap required, opened a new log
I20260928 21:17:20.567080 13229 ts_tablet_manager.cc:1397] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.567436 13229 raft_consensus.cc:348] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.567533 13229 raft_consensus.cc:374] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.567562 13229 raft_consensus.cc:729] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.567677 13229 consensus_queue.cc:260] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.567910 13229 ts_tablet_manager.cc:1428] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.568109 13235 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.568214 13235 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.561534 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a8793760114adebaf276ee1348989c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.571141 13227 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: Bootstrap starting.
I20260928 21:17:20.571321 13228 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: Bootstrap starting.
I20260928 21:17:20.571967 13227 tablet_bootstrap.cc:654] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.572257 13228 tablet_bootstrap.cc:654] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.573091 13227 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: No bootstrap required, opened a new log
I20260928 21:17:20.573171 13227 ts_tablet_manager.cc:1397] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.573416 13228 tablet_bootstrap.cc:492] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: No bootstrap required, opened a new log
I20260928 21:17:20.573498 13228 ts_tablet_manager.cc:1397] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:17:20.573537 13227 raft_consensus.cc:348] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.573670 13227 raft_consensus.cc:374] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.573705 13227 raft_consensus.cc:729] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431de377836a44b99c60264008f9d483, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.573817 13227 consensus_queue.cc:260] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.573896 13228 raft_consensus.cc:348] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.574008 13228 raft_consensus.cc:374] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.574045 13228 raft_consensus.cc:729] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.574074 13227 ts_tablet_manager.cc:1428] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.574156 13228 consensus_queue.cc:260] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.574292 13234 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.574394 13228 ts_tablet_manager.cc:1428] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.574407 13234 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.574573 13233 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.574733 13233 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.626682 13233 raft_consensus.cc:482] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:20.626829 13233 raft_consensus.cc:504] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.628031 13233 leader_election.cc:288] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:20.632817 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.628781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:34911 (local address 127.5.18.193:45709)
0928 21:17:20.628908 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.628912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.628926 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:20.628977 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.628980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.629028 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.629096 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.629538 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.630430 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.631917 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.632312 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.632417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.632470 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.632631 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.632706 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:17:20.632817 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.628251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45709 (local address 127.5.18.195:34911)
0928 21:17:20.628527 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.628542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.628626 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.629255 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.629258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.629279 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.629396 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.629403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.630821 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.630825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.631798 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.631808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.632417 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.632421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.632524 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.632546 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.632608 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.632681 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 21:17:20.633263 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a8793760114adebaf276ee1348989c" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483" is_pre_election: true
I20260928 21:17:20.633422 12873 raft_consensus.cc:2454] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:20.633805 13084 leader_election.cc:302] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431de377836a44b99c60264008f9d483, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:20.633857 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.628663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:57209 (local address 127.5.18.194:38643)
0928 21:17:20.628765 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.628770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.629055 (+ 285us) server_negotiation.cc:407] Connection header received
0928 21:17:20.629220 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.629223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.629264 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.629304 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:20.630157 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.630935 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.632475 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.633049 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.633137 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.633184 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.633665 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.633737 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:20.634065 13233 raft_consensus.cc:2781] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:20.634122 13233 raft_consensus.cc:482] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:20.634177 13233 raft_consensus.cc:3037] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.634276 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.628317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38643 (local address 127.5.18.195:57209)
0928 21:17:20.628976 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.628989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.629119 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.629430 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.629432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.629446 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:20.630016 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.630026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.631092 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.631095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.632318 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.632336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.633424 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.633427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.633526 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.633545 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.634020 (+ 475us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.634093 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":90,"threads_started":1}
I20260928 21:17:20.634774 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a8793760114adebaf276ee1348989c" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153" is_pre_election: true
I20260928 21:17:20.635011 13003 raft_consensus.cc:2454] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:20.635387 13233 raft_consensus.cc:504] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.635720 13233 leader_election.cc:288] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Requested vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:20.636062 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a8793760114adebaf276ee1348989c" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483"
I20260928 21:17:20.636168 12873 raft_consensus.cc:3037] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.636953 12873 raft_consensus.cc:2454] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:20.637187 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a8793760114adebaf276ee1348989c" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153"
I20260928 21:17:20.637212 13084 leader_election.cc:302] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431de377836a44b99c60264008f9d483, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:20.637293 13003 raft_consensus.cc:3037] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:20.637373 13233 raft_consensus.cc:2781] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:20.637583 13233 raft_consensus.cc:686] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [term 1 LEADER]: Becoming Leader. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Running, Role: LEADER
I20260928 21:17:20.637822 13233 consensus_queue.cc:237] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.638411 13003 raft_consensus.cc:2454] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:20.639104 13266 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: TxnStatusTablet state changed. Reason: New leader dd69b0a60023499689e1c3cfccaa9554. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.639263 13266 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:17:20.639529 13268 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:17:20.639654 13268 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:17:20.640110 12715 catalog_manager.cc:5495] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } health_report { overall_health: HEALTHY } } }
I20260928 21:17:20.670446 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.666619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.5.18.195:38115)
0928 21:17:20.666729 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.666733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.666869 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:17:20.667077 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.667079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.667135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.667218 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:20.667444 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.667928 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.669107 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.669316 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.669392 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.669708 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.670123 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.670206 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.670289 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.670335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:17:20.670511 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.666538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:38115 (local address 127.0.0.1:49018)
0928 21:17:20.666788 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.666802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.666921 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.667241 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.667243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.667252 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:17:20.667364 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.667370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.668013 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.668016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.668992 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.669002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.669409 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.669412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.669503 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.669555 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.670201 (+ 646us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.670212 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.670280 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.670371 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 21:17:20.673290 12873 raft_consensus.cc:1261] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.673410 13003 raft_consensus.cc:1261] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.673794 13233 consensus_queue.cc:1035] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.673955 13266 consensus_queue.cc:1035] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.675372 13234 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: TxnStatusTablet state changed. Reason: New leader dd69b0a60023499689e1c3cfccaa9554. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.675483 13234 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.676034 13235 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: TxnStatusTablet state changed. Reason: New leader dd69b0a60023499689e1c3cfccaa9554. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.676137 13235 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.676939 13241 mvcc.cc:205] Tried to move back new op lower bound from 7334421465792864256 to 7334421465655128064. Current Snapshot: MvccSnapshot[applied={T|T < 7334421465792864256}]
I20260928 21:17:20.677884 13233 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.677983 13233 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:17:20.679030 13234 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.679131 13234 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P 431de377836a44b99c60264008f9d483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.680007 13266 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.680102 13266 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P dd69b0a60023499689e1c3cfccaa9554: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:17:20.680609 13235 tablet_replica.cc:439] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } } }
I20260928 21:17:20.680707 13235 tablet_replica.cc:442] T 61a8793760114adebaf276ee1348989c P c94d26053e0647d881d0118cc12d0153: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:17:20.680914 13240 mvcc.cc:205] Tried to move back new op lower bound from 7334421465792864256 to 7334421465655128064. Current Snapshot: MvccSnapshot[applied={T|T < 7334421465792864256}]
I20260928 21:17:20.690905 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.687707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:37307 (local address 127.5.18.195:38115)
0928 21:17:20.687810 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.687814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.687981 (+ 167us) server_negotiation.cc:407] Connection header received
0928 21:17:20.688096 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.688099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.688153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.688210 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:20.688428 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.688913 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.689862 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.690129 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.690212 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.690483 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.690582 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.690642 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.690730 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.690776 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:17:20.690925 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.687643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:38115 (local address 127.5.18.194:37307)
0928 21:17:20.687910 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.687921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.688018 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.688227 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.688230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.688238 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:17:20.688342 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.688347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.689009 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.689012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.689752 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.689761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.690224 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.690228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.690324 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.690369 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.690655 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.690662 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.690720 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.690806 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":98,"threads_started":1}
I20260928 21:17:20.697198 13153 raft_consensus.cc:1261] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Refusing update from remote peer c94d26053e0647d881d0118cc12d0153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.697219 12873 raft_consensus.cc:1261] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Refusing update from remote peer c94d26053e0647d881d0118cc12d0153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:20.697681 13235 consensus_queue.cc:1035] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.697834 13237 consensus_queue.cc:1035] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:20.720532 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.716270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:48217 (local address 127.5.18.194:38643)
0928 21:17:20.716383 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.716388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.716597 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:17:20.716768 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.716771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.716822 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.716922 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:20.717415 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.718233 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.719464 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.719704 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.719800 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.720115 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.720190 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.720254 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.720348 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.720403 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:17:20.720784 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.716191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38643 (local address 127.5.18.195:48217)
0928 21:17:20.716503 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:20.716518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:20.716672 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:20.717027 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:20.717030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:20.717049 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:17:20.717293 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.717302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.718361 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.718364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.719359 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:20.719369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.719868 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.719870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:20.719951 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.719990 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:17:20.720262 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:17:20.720269 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:17:20.720337 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:17:20.720415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260928 21:17:20.743093 12715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:20.743237 12715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:20.745059 12715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Sending DeleteTablet for 3 replicas of tablet a235388fe78047299574ff1415e555a7
I20260928 21:17:20.745601 12853 tablet_service.cc:1514] Processing DeleteTablet for tablet a235388fe78047299574ff1415e555a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 21:17:20.745668 12993 tablet_service.cc:1514] Processing DeleteTablet for tablet a235388fe78047299574ff1415e555a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:46088
I20260928 21:17:20.745791 13251 tablet_replica.cc:330] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: stopping tablet replica
I20260928 21:17:20.745824 13249 tablet_replica.cc:330] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: stopping tablet replica
I20260928 21:17:20.745898 13251 raft_consensus.cc:2229] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:20.745908 13249 raft_consensus.cc:2229] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:20.745895 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet a235388fe78047299574ff1415e555a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:20 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49016
I20260928 21:17:20.745990 13251 raft_consensus.cc:2258] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.746045 13250 tablet_replica.cc:330] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: stopping tablet replica
I20260928 21:17:20.746093 13249 raft_consensus.cc:2258] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.746145 13250 raft_consensus.cc:2229] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:20.746235 13250 raft_consensus.cc:2258] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:20.746891 13249 ts_tablet_manager.cc:1905] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.746933 13250 ts_tablet_manager.cc:1905] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.749604 13250 ts_tablet_manager.cc:1918] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:20.749679 13250 log.cc:1196] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/a235388fe78047299574ff1415e555a7
I20260928 21:17:20.749718 13249 ts_tablet_manager.cc:1918] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:20.749788 13249 log.cc:1196] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/a235388fe78047299574ff1415e555a7
I20260928 21:17:20.749917 13250 ts_tablet_manager.cc:1939] T a235388fe78047299574ff1415e555a7 P dd69b0a60023499689e1c3cfccaa9554: Deleting consensus metadata
I20260928 21:17:20.750056 13249 ts_tablet_manager.cc:1939] T a235388fe78047299574ff1415e555a7 P c94d26053e0647d881d0118cc12d0153: Deleting consensus metadata
I20260928 21:17:20.750542 12701 catalog_manager.cc:4841] TS dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115): tablet a235388fe78047299574ff1415e555a7 (table test-table [id=055c32c7799045fa9916fa261077719f]) successfully deleted
I20260928 21:17:20.750664 12702 catalog_manager.cc:4841] TS c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643): tablet a235388fe78047299574ff1415e555a7 (table test-table [id=055c32c7799045fa9916fa261077719f]) successfully deleted
I20260928 21:17:20.751394 13251 ts_tablet_manager.cc:1905] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:20.754101 13251 ts_tablet_manager.cc:1918] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:20.754179 13251 log.cc:1196] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/a235388fe78047299574ff1415e555a7
I20260928 21:17:20.754400 13251 ts_tablet_manager.cc:1939] T a235388fe78047299574ff1415e555a7 P 431de377836a44b99c60264008f9d483: Deleting consensus metadata
I20260928 21:17:20.755060 12700 catalog_manager.cc:4841] TS 431de377836a44b99c60264008f9d483 (127.5.18.193:45709): tablet a235388fe78047299574ff1415e555a7 (table test-table [id=055c32c7799045fa9916fa261077719f]) successfully deleted
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:17:20.777108 12765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:20.766430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.5.18.254:40467)
0928 21:17:20.766610 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.766615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.769869 (+ 3254us) server_negotiation.cc:407] Connection header received
0928 21:17:20.770102 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.770105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.770154 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.770202 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.771145 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.771994 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.773171 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.773492 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.773611 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.776985 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47604: BlockingRecv error: recv got EOF from 127.0.0.1:47604 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 21:17:20 dist-test-slave-rc9f krb5kdc[12671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM
I20260928 21:17:20.788688 12765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:20.779252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.5.18.254:40467)
0928 21:17:20.779354 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:20.779357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:20.779865 (+ 508us) server_negotiation.cc:407] Connection header received
0928 21:17:20.780012 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:20.780015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:20.780056 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:20.780099 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:20.780863 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.781608 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.782696 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:20.783022 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:20.783116 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:20.785645 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:20.785649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:20.785652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:20.785676 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:20.787206 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.787614 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.787617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.787619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.787698 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:20.787924 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:20.787927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:20.787929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:20.788141 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:20.788269 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:20.788429 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:17:20.788537 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:17:20.791134 12715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:20.791603 12715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:20.796244 12853 tablet_service.cc:1467] Processing CreateTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (DEFAULT_TABLE table=test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.796317 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (DEFAULT_TABLE table=test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.796533 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b32279c98b4c5194c2a39d0ac5d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.796541 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b32279c98b4c5194c2a39d0ac5d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.796486 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (DEFAULT_TABLE table=test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:20.796749 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b32279c98b4c5194c2a39d0ac5d9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:20.798864 13229 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Bootstrap starting.
I20260928 21:17:20.798864 13227 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Bootstrap starting.
I20260928 21:17:20.798951 13228 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Bootstrap starting.
I20260928 21:17:20.799645 13227 tablet_bootstrap.cc:654] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.799644 13229 tablet_bootstrap.cc:654] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.799693 13228 tablet_bootstrap.cc:654] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:20.800909 13228 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: No bootstrap required, opened a new log
I20260928 21:17:20.800982 13228 ts_tablet_manager.cc:1397] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.801183 13227 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: No bootstrap required, opened a new log
I20260928 21:17:20.801251 13227 ts_tablet_manager.cc:1397] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:20.801374 13229 tablet_bootstrap.cc:492] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: No bootstrap required, opened a new log
I20260928 21:17:20.801319 13228 raft_consensus.cc:348] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.801414 13228 raft_consensus.cc:374] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.801440 13228 raft_consensus.cc:729] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.801445 13229 ts_tablet_manager.cc:1397] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:17:20.801541 13228 consensus_queue.cc:260] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.801796 13228 ts_tablet_manager.cc:1428] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.801856 13229 raft_consensus.cc:348] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.801856 13227 raft_consensus.cc:348] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.801956 13227 raft_consensus.cc:374] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.801955 13229 raft_consensus.cc:374] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:20.801985 13229 raft_consensus.cc:729] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.801985 13227 raft_consensus.cc:729] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431de377836a44b99c60264008f9d483, State: Initialized, Role: FOLLOWER
I20260928 21:17:20.802098 13227 consensus_queue.cc:260] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.802098 13229 consensus_queue.cc:260] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:20.802331 13229 ts_tablet_manager.cc:1428] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:20.802331 13227 ts_tablet_manager.cc:1428] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:21.093276 13273 raft_consensus.cc:482] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:21.093405 13273 raft_consensus.cc:504] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.093817 13273 leader_election.cc:288] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:21.094230 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b32279c98b4c5194c2a39d0ac5d9a3" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153" is_pre_election: true
I20260928 21:17:21.094231 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b32279c98b4c5194c2a39d0ac5d9a3" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483" is_pre_election: true
I20260928 21:17:21.094381 12873 raft_consensus.cc:2454] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:21.094383 13003 raft_consensus.cc:2454] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:21.094669 13086 leader_election.cc:302] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:21.094838 13273 raft_consensus.cc:2781] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:21.094900 13273 raft_consensus.cc:482] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:21.094926 13273 raft_consensus.cc:3037] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.095927 13273 raft_consensus.cc:504] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.096221 13273 leader_election.cc:288] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Requested vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:21.096596 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b32279c98b4c5194c2a39d0ac5d9a3" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153"
I20260928 21:17:21.096597 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98b32279c98b4c5194c2a39d0ac5d9a3" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483"
I20260928 21:17:21.096701 13003 raft_consensus.cc:3037] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.096725 12873 raft_consensus.cc:3037] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.097620 13003 raft_consensus.cc:2454] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:21.097635 12873 raft_consensus.cc:2454] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:21.097872 13086 leader_election.cc:302] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:21.098027 13273 raft_consensus.cc:2781] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:21.098158 13273 raft_consensus.cc:686] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 1 LEADER]: Becoming Leader. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Running, Role: LEADER
I20260928 21:17:21.098290 13273 consensus_queue.cc:237] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.099752 12716 catalog_manager.cc:5495] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } health_report { overall_health: HEALTHY } } }
I20260928 21:17:21.167580 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:21.164766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.5.18.195:38115)
0928 21:17:21.164906 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:21.164911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:21.164925 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:21.164972 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:21.164975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:21.165018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:21.165078 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:21.165295 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.165839 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.166658 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.166875 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.166957 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:21.167184 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:21.167247 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:21.167303 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:21.167408 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:17:21.167455 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:17:21.169556 12873 raft_consensus.cc:1261] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:21.169641 13003 raft_consensus.cc:1261] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:21.170032 13273 consensus_queue.cc:1035] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:21.170230 13266 consensus_queue.cc:1035] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:21.179430 12713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:21.179572 12713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:21.181389 12713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Sending DeleteTablet for 3 replicas of tablet 98b32279c98b4c5194c2a39d0ac5d9a3
I20260928 21:17:21.181880 12853 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 21:17:21.182003 12993 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:46088
I20260928 21:17:21.182121 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet 98b32279c98b4c5194c2a39d0ac5d9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49016
I20260928 21:17:21.182176 13251 tablet_replica.cc:330] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: stopping tablet replica
I20260928 21:17:21.182289 13251 raft_consensus.cc:2229] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:21.182412 13251 raft_consensus.cc:2258] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:21.182569 13249 tablet_replica.cc:330] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: stopping tablet replica
I20260928 21:17:21.182678 13249 raft_consensus.cc:2229] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:21.182790 13249 raft_consensus.cc:2258] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:21.182943 13250 tablet_replica.cc:330] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: stopping tablet replica
I20260928 21:17:21.183039 13250 raft_consensus.cc:2229] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:21.182991 12713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:17:21.183148 13251 ts_tablet_manager.cc:1905] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.183295 13250 raft_consensus.cc:2258] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:17:21.183337 12713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:21.184028 13250 ts_tablet_manager.cc:1905] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.185499 13251 ts_tablet_manager.cc:1918] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:21.185586 13251 log.cc:1196] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/98b32279c98b4c5194c2a39d0ac5d9a3
I20260928 21:17:21.185823 13251 ts_tablet_manager.cc:1939] T 98b32279c98b4c5194c2a39d0ac5d9a3 P 431de377836a44b99c60264008f9d483: Deleting consensus metadata
I20260928 21:17:21.186283 13250 ts_tablet_manager.cc:1918] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:21.186345 13250 log.cc:1196] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/98b32279c98b4c5194c2a39d0ac5d9a3
I20260928 21:17:21.186558 13250 ts_tablet_manager.cc:1939] T 98b32279c98b4c5194c2a39d0ac5d9a3 P dd69b0a60023499689e1c3cfccaa9554: Deleting consensus metadata
I20260928 21:17:21.187173 12701 catalog_manager.cc:4841] TS dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115): tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (table test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]) successfully deleted
I20260928 21:17:21.187824 12993 tablet_service.cc:1467] Processing CreateTablet for tablet 94a57c3057f247b1a8279e7a036967db (DEFAULT_TABLE table=test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:21.188001 13249 ts_tablet_manager.cc:1905] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.188025 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 94a57c3057f247b1a8279e7a036967db (DEFAULT_TABLE table=test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:21.188081 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a57c3057f247b1a8279e7a036967db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:21.188218 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a57c3057f247b1a8279e7a036967db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:21.190270 13228 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Bootstrap starting.
I20260928 21:17:21.190322 13229 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Bootstrap starting.
I20260928 21:17:21.191023 13228 tablet_bootstrap.cc:654] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:21.191030 13229 tablet_bootstrap.cc:654] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:21.190332 13249 ts_tablet_manager.cc:1918] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:21.191780 13249 log.cc:1196] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/98b32279c98b4c5194c2a39d0ac5d9a3
I20260928 21:17:21.192281 12700 catalog_manager.cc:4841] TS 431de377836a44b99c60264008f9d483 (127.5.18.193:45709): tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (table test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]) successfully deleted
I20260928 21:17:21.192404 13229 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: No bootstrap required, opened a new log
I20260928 21:17:21.192468 13229 ts_tablet_manager.cc:1397] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:21.192693 12853 tablet_service.cc:1467] Processing CreateTablet for tablet 94a57c3057f247b1a8279e7a036967db (DEFAULT_TABLE table=test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:21.192853 13229 raft_consensus.cc:348] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.192934 12853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94a57c3057f247b1a8279e7a036967db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:21.192953 13229 raft_consensus.cc:374] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:21.192986 13229 raft_consensus.cc:729] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94d26053e0647d881d0118cc12d0153, State: Initialized, Role: FOLLOWER
I20260928 21:17:21.193120 13229 consensus_queue.cc:260] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.193179 13228 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: No bootstrap required, opened a new log
I20260928 21:17:21.193243 13228 ts_tablet_manager.cc:1397] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:17:21.193383 13229 ts_tablet_manager.cc:1428] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:21.193615 13228 raft_consensus.cc:348] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.193711 13228 raft_consensus.cc:374] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:21.193740 13228 raft_consensus.cc:729] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Initialized, Role: FOLLOWER
I20260928 21:17:21.193837 13228 consensus_queue.cc:260] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.194060 13228 ts_tablet_manager.cc:1428] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:17:21.194367 13249 ts_tablet_manager.cc:1939] T 98b32279c98b4c5194c2a39d0ac5d9a3 P c94d26053e0647d881d0118cc12d0153: Deleting consensus metadata
I20260928 21:17:21.194927 12702 catalog_manager.cc:4841] TS c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643): tablet 98b32279c98b4c5194c2a39d0ac5d9a3 (table test-table [id=39f9f00076bd4bab9e26e731cacd0e2b]) successfully deleted
I20260928 21:17:21.195052 13227 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Bootstrap starting.
I20260928 21:17:21.195899 13227 tablet_bootstrap.cc:654] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:21.197129 13227 tablet_bootstrap.cc:492] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: No bootstrap required, opened a new log
I20260928 21:17:21.197201 13227 ts_tablet_manager.cc:1397] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:21.197595 13227 raft_consensus.cc:348] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.197695 13227 raft_consensus.cc:374] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:21.197731 13227 raft_consensus.cc:729] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431de377836a44b99c60264008f9d483, State: Initialized, Role: FOLLOWER
I20260928 21:17:21.197845 13227 consensus_queue.cc:260] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.198067 13227 ts_tablet_manager.cc:1428] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:17:21.247799 13273 raft_consensus.cc:482] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:21.247912 13273 raft_consensus.cc:504] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.248317 13273 leader_election.cc:288] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:21.248673 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a57c3057f247b1a8279e7a036967db" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483" is_pre_election: true
I20260928 21:17:21.248706 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a57c3057f247b1a8279e7a036967db" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153" is_pre_election: true
I20260928 21:17:21.248826 13003 raft_consensus.cc:2454] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:21.248826 12873 raft_consensus.cc:2454] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 0.
I20260928 21:17:21.249091 13084 leader_election.cc:302] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431de377836a44b99c60264008f9d483, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:21.249270 13273 raft_consensus.cc:2781] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:21.249346 13273 raft_consensus.cc:482] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:21.249390 13273 raft_consensus.cc:3037] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.250550 13273 raft_consensus.cc:504] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.250924 13273 leader_election.cc:288] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Requested vote from peers 431de377836a44b99c60264008f9d483 (127.5.18.193:45709), c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643)
I20260928 21:17:21.251245 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a57c3057f247b1a8279e7a036967db" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94d26053e0647d881d0118cc12d0153"
I20260928 21:17:21.251385 13003 raft_consensus.cc:3037] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.252477 13003 raft_consensus.cc:2454] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:21.252727 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94a57c3057f247b1a8279e7a036967db" candidate_uuid: "dd69b0a60023499689e1c3cfccaa9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431de377836a44b99c60264008f9d483"
I20260928 21:17:21.252745 13086 leader_election.cc:302] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c94d26053e0647d881d0118cc12d0153, dd69b0a60023499689e1c3cfccaa9554; no voters:
I20260928 21:17:21.252861 12873 raft_consensus.cc:3037] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:21.252943 13273 raft_consensus.cc:2781] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:21.253013 13273 raft_consensus.cc:686] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 1 LEADER]: Becoming Leader. State: Replica: dd69b0a60023499689e1c3cfccaa9554, State: Running, Role: LEADER
I20260928 21:17:21.253320 13273 consensus_queue.cc:237] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } }
I20260928 21:17:21.253945 12873 raft_consensus.cc:2454] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd69b0a60023499689e1c3cfccaa9554 in term 1.
I20260928 21:17:21.255584 12716 catalog_manager.cc:5495] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "dd69b0a60023499689e1c3cfccaa9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd69b0a60023499689e1c3cfccaa9554" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 38115 } health_report { overall_health: HEALTHY } } }
I20260928 21:17:21.275512 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:21.272464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:44509 (local address 127.5.18.195:38115)
0928 21:17:21.272571 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:21.272574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:21.272732 (+ 158us) server_negotiation.cc:407] Connection header received
0928 21:17:21.272843 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:21.272845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:21.272882 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:21.272938 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:21.273143 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.273661 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.274680 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.274874 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.274936 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:21.275170 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:21.275228 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:21.275272 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:21.275349 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:17:21.275385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:17:21.275604 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:21.272408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:38115 (local address 127.5.18.195:44509)
0928 21:17:21.272659 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:21.272673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:21.272770 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:21.272958 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:21.272961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:21.272967 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:17:21.273064 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:21.273069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.273752 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.273756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:21.274580 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:21.274590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:21.274965 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:21.274969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:21.275051 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:21.275080 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:17:21.275288 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:17:21.275292 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:17:21.275346 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:17:21.275420 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
I20260928 21:17:21.280962 12873 raft_consensus.cc:1261] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:21.281306 13003 raft_consensus.cc:1261] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Refusing update from remote peer dd69b0a60023499689e1c3cfccaa9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:21.281296 13273 consensus_queue.cc:1035] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431de377836a44b99c60264008f9d483" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:21.281684 13273 consensus_queue.cc:1035] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94d26053e0647d881d0118cc12d0153" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:21.311962 12716 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:21.312108 12716 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:21.313886 12716 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b57eb581cb3349edb44282c9b1140f27: Sending DeleteTablet for 3 replicas of tablet 94a57c3057f247b1a8279e7a036967db
I20260928 21:17:21.314344 12853 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a57c3057f247b1a8279e7a036967db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:59338
I20260928 21:17:21.314366 12993 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a57c3057f247b1a8279e7a036967db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:46088
I20260928 21:17:21.314508 13249 tablet_replica.cc:330] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: stopping tablet replica
I20260928 21:17:21.314520 13251 tablet_replica.cc:330] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: stopping tablet replica
I20260928 21:17:21.314610 13249 raft_consensus.cc:2229] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:21.314610 13251 raft_consensus.cc:2229] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:21.314700 13251 raft_consensus.cc:2258] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:21.314700 13249 raft_consensus.cc:2258] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:21.314894 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet 94a57c3057f247b1a8279e7a036967db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:21 UTC) from {username='oryx', principal='oryx/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49016
I20260928 21:17:21.315045 13250 tablet_replica.cc:330] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: stopping tablet replica
I20260928 21:17:21.315142 13250 raft_consensus.cc:2229] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:17:21.315302 13250 raft_consensus.cc:2258] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:21.315876 13250 ts_tablet_manager.cc:1905] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.315876 13251 ts_tablet_manager.cc:1905] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.318220 13251 ts_tablet_manager.cc:1918] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:21.318226 13250 ts_tablet_manager.cc:1918] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:21.318296 13250 log.cc:1196] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/94a57c3057f247b1a8279e7a036967db
I20260928 21:17:21.318296 13251 log.cc:1196] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/94a57c3057f247b1a8279e7a036967db
I20260928 21:17:21.318526 13251 ts_tablet_manager.cc:1939] T 94a57c3057f247b1a8279e7a036967db P 431de377836a44b99c60264008f9d483: Deleting consensus metadata
I20260928 21:17:21.318539 13250 ts_tablet_manager.cc:1939] T 94a57c3057f247b1a8279e7a036967db P dd69b0a60023499689e1c3cfccaa9554: Deleting consensus metadata
I20260928 21:17:21.319128 12700 catalog_manager.cc:4841] TS 431de377836a44b99c60264008f9d483 (127.5.18.193:45709): tablet 94a57c3057f247b1a8279e7a036967db (table test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]) successfully deleted
I20260928 21:17:21.319132 12701 catalog_manager.cc:4841] TS dd69b0a60023499689e1c3cfccaa9554 (127.5.18.195:38115): tablet 94a57c3057f247b1a8279e7a036967db (table test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]) successfully deleted
I20260928 21:17:21.319815 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12788
I20260928 21:17:21.320078 13249 ts_tablet_manager.cc:1905] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:21.322196 13249 ts_tablet_manager.cc:1918] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:17:21.322283 13249 log.cc:1196] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/94a57c3057f247b1a8279e7a036967db
I20260928 21:17:21.322504 13249 ts_tablet_manager.cc:1939] T 94a57c3057f247b1a8279e7a036967db P c94d26053e0647d881d0118cc12d0153: Deleting consensus metadata
I20260928 21:17:21.323182 12702 catalog_manager.cc:4841] TS c94d26053e0647d881d0118cc12d0153 (127.5.18.194:38643): tablet 94a57c3057f247b1a8279e7a036967db (table test-table [id=b52a4f7d32944a2aab2a8fac4edf103e]) successfully deleted
I20260928 21:17:21.328498 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12928
I20260928 21:17:21.335866 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13068
I20260928 21:17:21.341303 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 12684
2026-09-28T21:17:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3091 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:21 dist-test-slave-rc9f krb5kdc[13315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:21 dist-test-slave-rc9f krb5kdc[13315](info): set up 2 sockets
Sep 28 21:17:21 dist-test-slave-rc9f krb5kdc[13315](info): commencing operation
krb5kdc: starting...
W20260928 21:17:22.285876 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:22 dist-test-slave-rc9f krb5kdc[13315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:22.395658 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44739
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:44739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:22.498987 13328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:22.499236 13328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:22.499289 13328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:22.503151 13328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:22.503202 13328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:22.503229 13328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:22.503252 13328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:22.503279 13328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:22.508268 13328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:44739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:22.509213 13328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:22.510187 13328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:22.517990 13334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:22.518010 13333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:22.518121 13328 server_base.cc:1034] running on GCE node
W20260928 21:17:22.517968 13336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:22.518707 13328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:22.519726 13328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:22.520902 13328 hybrid_clock.cc:648] HybridClock initialized: now 1790630242520881 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:17:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1191 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:22 dist-test-slave-rc9f krb5kdc[13341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:22 dist-test-slave-rc9f krb5kdc[13341](info): set up 2 sockets
Sep 28 21:17:22 dist-test-slave-rc9f krb5kdc[13341](info): commencing operation
krb5kdc: starting...
W20260928 21:17:23.477854 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:23 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:23.586042 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38459
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:23.690433 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:23.690685 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:23.690737 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:23.694625 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:23.694677 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:23.694705 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:23.694723 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:23.694744 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:23.699908 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:23.700879 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:23.701867 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:23.709733 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:23.709760 13354 server_base.cc:1034] running on GCE node
W20260928 21:17:23.709707 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:23.709712 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:23.710470 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:23.711474 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:23.712642 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790630243712632 us; error 34 us; skew 500 ppm
Sep 28 21:17:23 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:23.715126 13354 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:23.715900 13354 webserver.cc:460] Webserver started at http://127.5.18.254:45301/ using document root <none> and password file <none>
I20260928 21:17:23.716391 13354 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:23.716447 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:23.716646 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:23.718583 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "1457e1145853426a8be561e56011ec67"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "94c26d42db5df477d9f84442206cad8c"
server_key_iv: "3e7d56b5a27d475fea07c4afaf1c57b7"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.719110 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "1457e1145853426a8be561e56011ec67"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "94c26d42db5df477d9f84442206cad8c"
server_key_iv: "3e7d56b5a27d475fea07c4afaf1c57b7"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.722508 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:23.724622 13369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:23.725772 13354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:23.725886 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "1457e1145853426a8be561e56011ec67"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "94c26d42db5df477d9f84442206cad8c"
server_key_iv: "3e7d56b5a27d475fea07c4afaf1c57b7"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.725970 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:23.737190 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:23.739590 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:23.739724 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:23.745672 13354 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38459
I20260928 21:17:23.745718 13421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38459 every 8 connection(s)
I20260928 21:17:23.746529 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:23.749029 13422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:23.750608 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13354
I20260928 21:17:23.750746 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:23.751020 5195 external_mini_cluster.cc:1428] Setting key bee84768f177de5df3d26e680a4687a6
I20260928 21:17:23.755324 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Bootstrap starting.
I20260928 21:17:23.757746 13422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:23.758554 13422 log.cc:824] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:17:23 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:23.760285 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: No bootstrap required, opened a new log
I20260928 21:17:23.764492 13422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:23.764739 13422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:23.764739 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:23.752272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.5.18.254:38459)
0928 21:17:23.752675 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:23.752694 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:23.752733 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:17:23.753515 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:23.753537 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:23.753964 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:23.754310 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:23.754885 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:23.756267 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:23.757085 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:23.757774 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:23.757930 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:23.760087 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:23.760093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:23.760108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:23.760136 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:23.762015 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:23.762379 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:23.762385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:23.762392 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:23.762531 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:23.762735 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:23.762738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:23.762740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:23.763197 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:23.763375 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:23.763543 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 21:17:23.763837 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":96,"threads_started":1}
I20260928 21:17:23.764818 13422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1457e1145853426a8be561e56011ec67, State: Initialized, Role: FOLLOWER
I20260928 21:17:23.765218 13422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:23.765345 13422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:23.765407 13422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:23.765496 13422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:23.766268 13422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:23.766570 13422 leader_election.cc:302] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1457e1145853426a8be561e56011ec67; no voters:
I20260928 21:17:23.766827 13422 leader_election.cc:288] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:23.766896 13427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:23.767112 13427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 LEADER]: Becoming Leader. State: Replica: 1457e1145853426a8be561e56011ec67, State: Running, Role: LEADER
I20260928 21:17:23.767418 13427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:23.767865 13422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:23.768949 13428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1457e1145853426a8be561e56011ec67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } } }
I20260928 21:17:23.769049 13428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:23.769408 13436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:23.769373 13429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1457e1145853426a8be561e56011ec67. Latest consensus state: current_term: 1 leader_uuid: "1457e1145853426a8be561e56011ec67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } } }
I20260928 21:17:23.770000 13429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:23.771947 13436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:23.777036 13436 catalog_manager.cc:1347] Generated new cluster ID: ebdbf5f155e64af6a22d522e9348a72c
I20260928 21:17:23.777097 13436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:23.798911 13436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:23.800148 13436 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:23.813694 13436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Generated new TSK 0
I20260928 21:17:23.814337 13436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:23.842540 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:23.948694 13448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:23.948936 13448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:23.948976 13448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:23.952870 13448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:23.952925 13448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:23.952979 13448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:23.958029 13448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:23.959246 13448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:23.960186 13448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:23.968966 13454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:23.969015 13453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:23.968976 13456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:23.969247 13448 server_base.cc:1034] running on GCE node
I20260928 21:17:23.969609 13448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:23.970219 13448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:23.971364 13448 hybrid_clock.cc:648] HybridClock initialized: now 1790630243971347 us; error 30 us; skew 500 ppm
Sep 28 21:17:23 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:23.973869 13448 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:23.974628 13448 webserver.cc:460] Webserver started at http://127.5.18.193:42201/ using document root <none> and password file <none>
I20260928 21:17:23.975131 13448 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:23.975196 13448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:23.975373 13448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:23.977286 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "9a4e9d8162334d7d9eb298c24d5e92fc"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "a5694e1c6950a47efefbb7d245c7d528"
server_key_iv: "302babeecd35ab800fd7ef5d87fd72db"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.977825 13448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "9a4e9d8162334d7d9eb298c24d5e92fc"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "a5694e1c6950a47efefbb7d245c7d528"
server_key_iv: "302babeecd35ab800fd7ef5d87fd72db"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.981300 13448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 21:17:23.983598 13463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:23.984730 13448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:23.984829 13448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "9a4e9d8162334d7d9eb298c24d5e92fc"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "a5694e1c6950a47efefbb7d245c7d528"
server_key_iv: "302babeecd35ab800fd7ef5d87fd72db"
server_key_version: "encryptionkey@0"
I20260928 21:17:23.984916 13448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:24.002578 13448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:24.005276 13448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:24.005447 13448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:24.006033 13448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:24.006801 13448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:24.006846 13448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.006896 13448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:24.006925 13448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.015266 13448 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:33385
I20260928 21:17:24.015293 13576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:33385 every 8 connection(s)
I20260928 21:17:24.016085 13448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:24.017755 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13448
I20260928 21:17:24.017874 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:24.018167 5195 external_mini_cluster.cc:1428] Setting key 8f436436437a8e54d4d19df86fedff02
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:24.030001 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.017727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:60337 (local address 127.5.18.254:38459)
0928 21:17:24.017875 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.017879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.018735 (+ 856us) server_negotiation.cc:407] Connection header received
0928 21:17:24.020168 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.020170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.020210 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.020273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.021561 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.022327 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.023899 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.024120 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.024195 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.026820 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:24.026823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:24.026825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:24.026847 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:24.028134 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.028644 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.028647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.028648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.028720 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.028965 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.028967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.028968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.029126 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:24.029228 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:24.029713 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 21:17:24.029837 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:17:24.030802 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.018131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:60337)
0928 21:17:24.018593 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.018634 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.019959 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.020444 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.020453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.020895 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.021434 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.021449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.022448 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.022452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.023787 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.023798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.024261 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.024265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.024457 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.025013 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.025044 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.026660 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:24.028235 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.028239 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.028251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.028559 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.028782 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.028784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.028785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.028899 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.029229 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:24.029234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:24.029446 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 21:17:24.029669 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:17:24.029983 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260928 21:17:24.031997 13577 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38459
I20260928 21:17:24.032226 13577 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:24.032722 13577 heartbeater.cc:511] Master 127.5.18.254:38459 requested a full tablet report, sending...
I20260928 21:17:24.034394 13386 ts_manager.cc:194] Registered new tserver with Master: 9a4e9d8162334d7d9eb298c24d5e92fc (127.5.18.193:33385)
I20260928 21:17:24.035913 13386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:60337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:24.043954 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:24.150000 13582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:24.150269 13582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:24.150313 13582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:24.154282 13582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:24.154349 13582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:24.154408 13582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:24.159550 13582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:24.160513 13582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:24.161446 13582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:24.170538 13588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.170548 13590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.170550 13587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:24.170732 13582 server_base.cc:1034] running on GCE node
I20260928 21:17:24.171185 13582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:24.171777 13582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:24.172926 13582 hybrid_clock.cc:648] HybridClock initialized: now 1790630244172909 us; error 32 us; skew 500 ppm
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:24.175413 13582 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:24.176206 13582 webserver.cc:460] Webserver started at http://127.5.18.194:40833/ using document root <none> and password file <none>
I20260928 21:17:24.176708 13582 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:24.176770 13582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:24.176959 13582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:24.178960 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "56bac69c0fc3443b94687b38567df0b1"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "dcd5f5a465d10bcffe7239f549f7443c"
server_key_iv: "28de8296907eb5496e567b856863521d"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.179469 13582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "56bac69c0fc3443b94687b38567df0b1"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "dcd5f5a465d10bcffe7239f549f7443c"
server_key_iv: "28de8296907eb5496e567b856863521d"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.183001 13582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:24.185161 13597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.186290 13582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:24.186398 13582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "56bac69c0fc3443b94687b38567df0b1"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "dcd5f5a465d10bcffe7239f549f7443c"
server_key_iv: "28de8296907eb5496e567b856863521d"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.186482 13582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:24.210230 13582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:24.212867 13582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:24.213035 13582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:24.213606 13582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:24.214377 13582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:24.214423 13582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.214473 13582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:24.214501 13582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.222545 13582 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:42839
I20260928 21:17:24.222574 13710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:42839 every 8 connection(s)
I20260928 21:17:24.223384 13582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:24.229215 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13582
I20260928 21:17:24.229351 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:24.229646 5195 external_mini_cluster.cc:1428] Setting key f6ffdf8e4ffb21e5d45813df63dd6e16
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:24.236543 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.225034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:38857 (local address 127.5.18.254:38459)
0928 21:17:24.225184 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.225189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.226034 (+ 845us) server_negotiation.cc:407] Connection header received
0928 21:17:24.227258 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.227261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.227306 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.227372 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.228491 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.229092 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.230503 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.230770 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.230866 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.233316 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:24.233318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:24.233320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:24.233343 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:24.234698 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.235153 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.235157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.235158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.235239 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.235532 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.235536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.235538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.235714 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:24.235797 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:24.236264 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 21:17:24.236383 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:17:24.237313 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.225390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:38857)
0928 21:17:24.225902 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.225946 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.227076 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.227520 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.227546 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.227883 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.228377 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.228392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.229201 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.229203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.230326 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.230380 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.230921 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.230925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.231098 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.231641 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.231657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.233163 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:24.234799 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.234805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.234820 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.235063 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.235317 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.235320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.235322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.235451 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.235807 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:24.235814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:24.236049 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:17:24.236214 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 21:17:24.236521 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":155,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:24.238395 13711 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38459
I20260928 21:17:24.238603 13711 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:24.239065 13711 heartbeater.cc:511] Master 127.5.18.254:38459 requested a full tablet report, sending...
I20260928 21:17:24.239996 13386 ts_manager.cc:194] Registered new tserver with Master: 56bac69c0fc3443b94687b38567df0b1 (127.5.18.194:42839)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:24.240675 13386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:38857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:24.254042 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:24.359598 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:24.359864 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:24.359905 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:24.363852 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:24.363909 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:24.363963 13716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:24.369122 13716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:24.370105 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:24.371014 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:17:24.379937 13716 server_base.cc:1034] running on GCE node
W20260928 21:17:24.379895 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.379927 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.379895 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:24.380450 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:24.381029 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:24.382176 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790630244382153 us; error 32 us; skew 500 ppm
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:24.384707 13716 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:24.385497 13716 webserver.cc:460] Webserver started at http://127.5.18.195:45111/ using document root <none> and password file <none>
I20260928 21:17:24.386018 13716 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:24.386081 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:24.386282 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:24.388324 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "dbc15bfe54494474b1dc7988b989a239"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "bd3978cc51273024d7e79c05aa4409fe"
server_key_iv: "a45ad55a30913a7968ed70c398c1c37b"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.388875 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "dbc15bfe54494474b1dc7988b989a239"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "bd3978cc51273024d7e79c05aa4409fe"
server_key_iv: "a45ad55a30913a7968ed70c398c1c37b"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.392416 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:24.394695 13731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.395850 13716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:24.395951 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "dbc15bfe54494474b1dc7988b989a239"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "bd3978cc51273024d7e79c05aa4409fe"
server_key_iv: "a45ad55a30913a7968ed70c398c1c37b"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.396039 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:24.414064 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:24.417012 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:24.417186 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:24.417742 13716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:24.418512 13716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:24.418565 13716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.418613 13716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:24.418638 13716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.426440 13716 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:44973
I20260928 21:17:24.426476 13844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:44973 every 8 connection(s)
I20260928 21:17:24.427282 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:24.429270 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13716
I20260928 21:17:24.429373 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:24.429723 5195 external_mini_cluster.cc:1428] Setting key 971352e67b0d1a0efdcdb62f806e23d4
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:24.439232 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.428850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:41719 (local address 127.5.18.254:38459)
0928 21:17:24.428975 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.428979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.429939 (+ 960us) server_negotiation.cc:407] Connection header received
0928 21:17:24.431329 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.431333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.431384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.431460 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.432569 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.433076 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.434207 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.434425 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.434502 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.436624 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:24.436627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:24.436629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:24.436652 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:24.437740 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.438135 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.438138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.438139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.438203 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:24.438434 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:24.438436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:24.438437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:24.438587 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:24.438655 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:24.439026 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:17:24.439117 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:17:24.439996 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.429304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:41719)
0928 21:17:24.429784 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.429840 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.431144 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.431640 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.431649 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.431993 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.432458 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.432472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.433160 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.433162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.434082 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.434091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.434532 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.434535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.434664 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.435183 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.435199 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.436489 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:24.437821 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.437825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.437836 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.438061 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.438259 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:24.438261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:24.438262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:24.438375 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:24.438654 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:24.438658 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:24.438860 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 21:17:24.438998 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:17:24.439232 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":168,"threads_started":1}
I20260928 21:17:24.440973 13845 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38459
I20260928 21:17:24.441179 13845 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:24.441676 13845 heartbeater.cc:511] Master 127.5.18.254:38459 requested a full tablet report, sending...
I20260928 21:17:24.442627 13386 ts_manager.cc:194] Registered new tserver with Master: dbc15bfe54494474b1dc7988b989a239 (127.5.18.195:44973)
I20260928 21:17:24.443185 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:17:24.443300 13386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:41719
WARNING: no policy specified for oryx/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
Entry for principal oryx/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab.
WARNING: no policy specified for oryx/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
Entry for principal oryx/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab.
WARNING: no policy specified for oryx/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
Entry for principal oryx/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab.
I20260928 21:17:24.505070 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13448
I20260928 21:17:24.509747 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13582
I20260928 21:17:24.513877 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13716
I20260928 21:17:24.517946 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13354
I20260928 21:17:24.522852 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38459
--webserver_interface=127.5.18.254
--webserver_port=45301
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:24.627233 13854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:24.627476 13854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:24.627532 13854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:24.631457 13854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:24.631508 13854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:24.631533 13854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:24.631567 13854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:24.631594 13854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:24.637683 13854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:24.639055 13854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:24.640407 13854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:24.648535 13860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.648587 13862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.648571 13859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:24.648656 13854 server_base.cc:1034] running on GCE node
I20260928 21:17:24.649345 13854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:24.650391 13854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:24.651548 13854 hybrid_clock.cc:648] HybridClock initialized: now 1790630244651532 us; error 39 us; skew 500 ppm
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:24.654126 13854 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:24.654955 13854 webserver.cc:460] Webserver started at http://127.5.18.254:45301/ using document root <none> and password file <none>
I20260928 21:17:24.655455 13854 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:24.655531 13854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:24.659170 13854 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:24.661170 13869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.662343 13854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:24.662447 13854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "1457e1145853426a8be561e56011ec67"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "94c26d42db5df477d9f84442206cad8c"
server_key_iv: "3e7d56b5a27d475fea07c4afaf1c57b7"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.662952 13854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:24.694799 13854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:24.697917 13854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:24.698091 13854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:24.704608 13854 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38459
I20260928 21:17:24.704628 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38459 every 8 connection(s)
I20260928 21:17:24.705638 13854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:24.707993 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13854
I20260928 21:17:24.708372 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:33385
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:38459
--webserver_port=42201
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:24.709791 13922 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:17:24.711596 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Bootstrap starting.
I20260928 21:17:24.732968 13922 log.cc:824] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:24.740722 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:17:24.741088 13922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Bootstrap complete.
I20260928 21:17:24.746295 13922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:24.746786 13922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1457e1145853426a8be561e56011ec67, State: Initialized, Role: FOLLOWER
I20260928 21:17:24.747303 13922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:24.747439 13922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:24.747502 13922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:24.747599 13922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:17:24.749992 13922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:24.750337 13922 leader_election.cc:302] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1457e1145853426a8be561e56011ec67; no voters:
I20260928 21:17:24.750662 13922 leader_election.cc:288] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:17:24.750774 13926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:17:24.751031 13926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [term 2 LEADER]: Becoming Leader. State: Replica: 1457e1145853426a8be561e56011ec67, State: Running, Role: LEADER
I20260928 21:17:24.751353 13926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } }
I20260928 21:17:24.751865 13922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:24.752575 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1457e1145853426a8be561e56011ec67. Latest consensus state: current_term: 2 leader_uuid: "1457e1145853426a8be561e56011ec67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } } }
I20260928 21:17:24.752683 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:24.753352 13935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:24.753324 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1457e1145853426a8be561e56011ec67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457e1145853426a8be561e56011ec67" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38459 } } }
I20260928 21:17:24.753444 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:24.756453 13935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:24.757104 13935 catalog_manager.cc:1259] Loaded cluster ID: ebdbf5f155e64af6a22d522e9348a72c
I20260928 21:17:24.757166 13935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:24.759483 13935 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:24.760118 13935 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1457e1145853426a8be561e56011ec67: Loaded TSK: 0
I20260928 21:17:24.761304 13935 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:17:24.828238 13924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:24.828469 13924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:24.828523 13924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:24.832516 13924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:24.832571 13924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:24.832647 13924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:24.837638 13924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.193.keytab
--principal=oryx/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:33385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=42201
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:24.838522 13924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:24.839418 13924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:24.848246 13950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:24.848373 13949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:24.848392 13924 server_base.cc:1034] running on GCE node
W20260928 21:17:24.848556 13952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:24.848930 13924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:24.849516 13924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:24.850703 13924 hybrid_clock.cc:648] HybridClock initialized: now 1790630244850681 us; error 34 us; skew 500 ppm
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630244, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:24.853610 13924 init.cc:377] Logged in from keytab as oryx/127.5.18.193@KRBTEST.COM (short username oryx)
I20260928 21:17:24.854421 13924 webserver.cc:460] Webserver started at http://127.5.18.193:42201/ using document root <none> and password file <none>
I20260928 21:17:24.854931 13924 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:24.855000 13924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:24.858826 13924 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:24.860972 13959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.862334 13924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:17:24.862443 13924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "9a4e9d8162334d7d9eb298c24d5e92fc"
format_stamp: "Formatted at 2026-09-28 21:17:23 on dist-test-slave-rc9f"
server_key: "a5694e1c6950a47efefbb7d245c7d528"
server_key_iv: "302babeecd35ab800fd7ef5d87fd72db"
server_key_version: "encryptionkey@0"
I20260928 21:17:24.862978 13924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:24.879917 13924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:24.882864 13924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:24.883029 13924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:24.883563 13924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:24.884342 13924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:24.884390 13924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.884440 13924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:24.884465 13924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:24.893306 13924 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:33385
I20260928 21:17:24.893355 14072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:33385 every 8 connection(s)
I20260928 21:17:24.894294 13924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:24.895048 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 13924
I20260928 21:17:24.895423 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:42839
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:38459
--webserver_port=40833
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:17:24.909989 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:24.898060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57305 (local address 127.5.18.254:38459)
0928 21:17:24.898641 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.898659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.898788 (+ 129us) server_negotiation.cc:407] Connection header received
0928 21:17:24.900566 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.900578 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.900854 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.901125 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.902736 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.903707 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.905354 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.905867 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.905980 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.909274 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:57305: BlockingRecv error: recv got EOF from 127.5.18.193:57305 (error 108)
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":122,"threads_started":1}
I20260928 21:17:24.909996 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.898100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:57305)
0928 21:17:24.898617 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.898674 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.900176 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.901345 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.901355 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.901874 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.902546 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.902568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.903839 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.903842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.905244 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.905256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.906004 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.906008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.906192 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.906773 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.906792 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.909045 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
W20260928 21:17:24.910413 14073 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:38459 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.18.254:38459: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:24.915215 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.910812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:46883)
0928 21:17:24.910906 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.910920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.910990 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.911183 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.911186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.911321 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.911400 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.911405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.912091 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.912094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.912616 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.912623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.913004 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.913007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.913097 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.913475 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.913486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.915092 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:17:24.915283 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:24.910886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:46883 (local address 127.5.18.254:38459)
0928 21:17:24.911006 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.911010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.911024 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:17:24.911057 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.911059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.911098 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.911160 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.911496 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.911989 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.912715 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.912916 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.912995 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.915198 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:46883: BlockingRecv error: recv got EOF from 127.5.18.193:46883 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:24 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:24.919718 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:24.915641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:35759)
0928 21:17:24.915732 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:24.915746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:24.915814 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:24.916008 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:24.916011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:24.916156 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:24.916217 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.916222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.916863 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.916866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.917361 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:24.917367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.917794 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.917797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:24.917868 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.918246 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:24.918256 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:24.919613 (+ 1357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 21:17:24.919811 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:24.915701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:35759 (local address 127.5.18.254:38459)
0928 21:17:24.915788 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:24.915792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:24.915803 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:24.915889 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:24.915892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:24.915929 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:24.915986 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:24.916313 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.916772 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.917455 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:24.917696 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:24.917783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:24.919724 (+ 1941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:35759: BlockingRecv error: recv got EOF from 127.5.18.193:35759 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 21:17:24.919981 14073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:17:25.005604 14075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:25.005877 14075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:25.005919 14075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:25.009879 14075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:25.009938 14075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:25.009995 14075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:25.015172 14075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.194.keytab
--principal=oryx/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:42839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=40833
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:25.016160 14075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:25.017096 14075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:17:25.026094 14075 server_base.cc:1034] running on GCE node
W20260928 21:17:25.026106 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:25.026062 14083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:25.026055 14085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:25.026633 14075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:25.027208 14075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:25.028362 14075 hybrid_clock.cc:648] HybridClock initialized: now 1790630245028342 us; error 29 us; skew 500 ppm
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630245, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:25.030838 14075 init.cc:377] Logged in from keytab as oryx/127.5.18.194@KRBTEST.COM (short username oryx)
I20260928 21:17:25.031633 14075 webserver.cc:460] Webserver started at http://127.5.18.194:40833/ using document root <none> and password file <none>
I20260928 21:17:25.032145 14075 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:25.032222 14075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:25.035802 14075 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:17:25.037667 14092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.038787 14075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:25.038887 14075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "56bac69c0fc3443b94687b38567df0b1"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "dcd5f5a465d10bcffe7239f549f7443c"
server_key_iv: "28de8296907eb5496e567b856863521d"
server_key_version: "encryptionkey@0"
I20260928 21:17:25.039368 14075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:25.049305 14075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:25.051856 14075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:25.052006 14075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:25.052574 14075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:25.053345 14075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:25.053398 14075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.053464 14075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:25.053493 14075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.061399 14075 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:42839
I20260928 21:17:25.061421 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:42839 every 8 connection(s)
I20260928 21:17:25.062373 14075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.071997 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14075
W20260928 21:17:25.072216 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.063944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:33671 (local address 127.5.18.254:38459)
0928 21:17:25.064073 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.064078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.064897 (+ 819us) server_negotiation.cc:407] Connection header received
0928 21:17:25.066130 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.066133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.066184 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.066278 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.067459 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.067969 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.068994 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.069196 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.069264 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.072111 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:33671: BlockingRecv error: recv got EOF from 127.5.18.194:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:25.072456 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:44973
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:38459
--webserver_port=45111
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:17:25.072914 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.064354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:33671)
0928 21:17:25.064759 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.064798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.065943 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.066435 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.066441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.066826 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.067336 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.067353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.068067 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.068069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.068908 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.068917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.069297 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.069299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.069427 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.069921 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.069936 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.071989 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
W20260928 21:17:25.073354 14206 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:38459 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.18.254:38459: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.078584 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.073784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:56885)
0928 21:17:25.073896 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.073909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.073980 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.074188 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.074190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.074307 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.074387 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.074391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.075267 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.075269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.075789 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.075795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.076309 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.076311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.076374 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.076825 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.076834 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.078496 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:17:25.078752 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.073862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:56885 (local address 127.5.18.254:38459)
0928 21:17:25.073970 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.073975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.073988 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:25.074047 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.074049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.074099 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.074173 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.074474 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.075183 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.075884 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.076210 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.076315 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.078648 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:56885: BlockingRecv error: recv got EOF from 127.5.18.194:56885 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.083580 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.078990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:45453)
0928 21:17:25.079077 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.079091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.079158 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.079362 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.079363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.079471 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.079531 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.079535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.080432 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.080434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.080917 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.080922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.081423 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.081426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.081476 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.081908 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.081918 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.083496 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 21:17:25.083732 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.079059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45453 (local address 127.5.18.254:38459)
0928 21:17:25.079181 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.079185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.079200 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:17:25.079233 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.079235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.079276 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.079355 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.079622 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.080344 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.081009 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.081331 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.081434 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.083630 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:45453: BlockingRecv error: recv got EOF from 127.5.18.194:45453 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:17:25.083762 14206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:17:25.181527 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:25.181823 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:25.181865 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:25.185731 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:25.185783 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:25.185850 14209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:25.190786 14209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/krb5kdc/oryx_127.5.18.195.keytab
--principal=oryx/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:44973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=45111
--tserver_master_addrs=127.5.18.254:38459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:25.191731 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:25.192675 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:25.201452 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:25.201467 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:25.201467 14217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:25.201653 14209 server_base.cc:1034] running on GCE node
I20260928 21:17:25.202208 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:25.202803 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:25.203972 14209 hybrid_clock.cc:648] HybridClock initialized: now 1790630245203957 us; error 24 us; skew 500 ppm
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630245, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:25.206676 14209 init.cc:377] Logged in from keytab as oryx/127.5.18.195@KRBTEST.COM (short username oryx)
I20260928 21:17:25.207526 14209 webserver.cc:460] Webserver started at http://127.5.18.195:45111/ using document root <none> and password file <none>
I20260928 21:17:25.208012 14209 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:25.208094 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:25.211813 14209 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 21:17:25.213737 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.214886 14209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:25.214982 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "dbc15bfe54494474b1dc7988b989a239"
format_stamp: "Formatted at 2026-09-28 21:17:24 on dist-test-slave-rc9f"
server_key: "bd3978cc51273024d7e79c05aa4409fe"
server_key_iv: "a45ad55a30913a7968ed70c398c1c37b"
server_key_version: "encryptionkey@0"
I20260928 21:17:25.215500 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:25.229643 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:25.232646 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:25.232807 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:25.233326 14209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:25.234150 14209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:25.234197 14209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.234263 14209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:25.234287 14209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:25.241897 14209 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:44973
I20260928 21:17:25.241921 14337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:44973 every 8 connection(s)
I20260928 21:17:25.242828 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:25.248019 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14209
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:17:25.253737 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.244484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:42719 (local address 127.5.18.254:38459)
0928 21:17:25.244605 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.244608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.245427 (+ 819us) server_negotiation.cc:407] Connection header received
0928 21:17:25.246722 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.246724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.246772 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.246848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.247978 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.248542 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.249966 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.250265 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.250341 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.253612 (+ 3271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:42719: BlockingRecv error: recv got EOF from 127.5.18.195:42719 (error 108)
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":42}
I20260928 21:17:25.254444 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.244839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:42719)
0928 21:17:25.245280 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.245329 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.246529 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.246997 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.247003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.247345 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.247822 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.247843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.248710 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.248713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.249855 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.249865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.250399 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.250402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.250582 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.251166 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.251182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.253491 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
W20260928 21:17:25.254882 14338 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:38459 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.18.254:38459: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
I20260928 21:17:25.257783 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.248932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.5.18.254:38459)
0928 21:17:25.249179 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.249183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.249196 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:25.249228 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.249231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.249270 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.249362 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.250047 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.250731 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.251643 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.251951 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.252032 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.253727 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:25.253733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:25.253749 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:25.253779 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:25.255799 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:25.256202 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:25.256208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:25.256214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:25.256351 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:25.256552 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:25.256554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:25.256556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:25.256967 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:25.257124 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:25.257392 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 21:17:25.257629 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.260020 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.255249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:55461)
0928 21:17:25.255394 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.255406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.255468 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.255664 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.255666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.255791 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.255882 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.255889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.256660 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.256663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.257275 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.257286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.257789 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.257792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.257876 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.258284 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.258294 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.259920 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 21:17:25.260113 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.255329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:55461 (local address 127.5.18.254:38459)
0928 21:17:25.255433 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.255437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.255450 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:25.255527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.255530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.255577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.255649 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.255977 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.256571 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.257370 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.257690 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.257783 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.260024 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:55461: BlockingRecv error: recv got EOF from 127.5.18.195:55461 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.264362 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.260541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:46521)
0928 21:17:25.260637 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.260649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.260710 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.260872 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.260873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.260979 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.261041 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.261045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.261740 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.261742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.262248 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.262255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.262601 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.262603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.262652 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.262969 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.262978 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.264285 (+ 1307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 21:17:25.264472 14077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.260616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46521 (local address 127.5.18.254:38459)
0928 21:17:25.260727 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.260731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.260742 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:25.260767 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.260769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.260797 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.260855 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.261149 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.261659 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.262334 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.262525 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.262609 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.264383 (+ 1774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:46521: BlockingRecv error: recv got EOF from 127.5.18.195:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 21:17:25.264528 14338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:25 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:25.925348 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:25.920512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:59945)
0928 21:17:25.920771 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:25.920784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:25.920870 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:25.921080 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:25.921082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:25.921258 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:25.921370 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.921376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.922070 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.922073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.922639 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:25.922646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.923035 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.923038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:25.923099 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.923536 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:25.923546 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:25.925206 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
W20260928 21:17:25.925444 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:25.920585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:59945 (local address 127.5.18.254:38459)
0928 21:17:25.920808 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:25.920811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:25.920826 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:17:25.920943 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:25.920945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:25.920987 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:25.921058 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:25.921451 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.921992 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.922737 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:25.922956 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:25.923027 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:25.925333 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:59945: BlockingRecv error: recv got EOF from 127.5.18.193:59945 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:26.089708 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:26.084292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:36851)
0928 21:17:26.084543 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:26.084557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:26.084673 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:26.084962 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:26.084965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:26.085157 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:26.085246 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.085262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.086020 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.086023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.086787 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.086799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.087337 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.087340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.087403 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.087844 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:26.087855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:26.089554 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
W20260928 21:17:26.089813 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:26.084355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:36851 (local address 127.5.18.254:38459)
0928 21:17:26.084496 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:26.084501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:26.084638 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:17:26.084799 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:26.084802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:26.084848 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:26.084931 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:26.085367 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.085921 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.086904 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.087228 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.087306 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.089704 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:36851: BlockingRecv error: recv got EOF from 127.5.18.194:36851 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:26.270222 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:26.265068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:41817)
0928 21:17:26.265327 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:26.265339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:26.265434 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:26.265670 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:26.265673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:26.265837 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:26.265938 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.265944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.266708 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.266710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.267319 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.267330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.267724 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.267726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.267788 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.268282 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:26.268295 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:26.270101 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260928 21:17:26.270365 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:26.265133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:41817 (local address 127.5.18.254:38459)
0928 21:17:26.265248 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:26.265252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:26.265398 (+ 146us) server_negotiation.cc:407] Connection header received
0928 21:17:26.265504 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:26.265506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:26.265551 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:26.265643 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:26.266031 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.266622 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.267422 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.267636 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.267707 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.270247 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:41817: BlockingRecv error: recv got EOF from 127.5.18.195:41817 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:26 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:26.930725 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:26.925994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:56063)
0928 21:17:26.926246 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:26.926259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:26.926355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:26.926565 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:26.926567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:26.926736 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:26.926829 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.926834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.927530 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.927532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.928080 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:26.928087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.928452 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.928454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:26.928510 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.928944 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:26.928955 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:26.930613 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
W20260928 21:17:26.930867 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:26.926069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56063 (local address 127.5.18.254:38459)
0928 21:17:26.926314 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:26.926317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:26.926330 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:26.926428 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:26.926430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:26.926472 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:26.926542 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:26.926910 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.927449 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.928170 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:26.928376 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:26.928443 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:26.930744 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:56063: BlockingRecv error: recv got EOF from 127.5.18.193:56063 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:27.095827 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:27.090487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:36323)
0928 21:17:27.090740 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:27.090753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:27.090871 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:27.091150 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:27.091154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:27.091351 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:27.091449 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.091456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.092235 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.092239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.092803 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.092811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.093272 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.093276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.093363 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.093902 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:27.093914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:27.095688 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
W20260928 21:17:27.095935 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:27.090583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:36323 (local address 127.5.18.254:38459)
0928 21:17:27.090726 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:27.090731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:27.090833 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:17:27.090964 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:27.090967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:27.091020 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:27.091091 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:27.091567 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.092123 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.092919 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.093150 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.093241 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.095834 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:36323: BlockingRecv error: recv got EOF from 127.5.18.194:36323 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:27.276043 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:27.270946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:40325)
0928 21:17:27.271236 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:27.271252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:27.271366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:27.271616 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:27.271619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:27.271811 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:27.271919 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.271925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.272647 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.272651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.273216 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.273224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.273662 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.273666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.273748 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.274212 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:27.274223 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:27.275905 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
W20260928 21:17:27.276136 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:27.271058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:40325 (local address 127.5.18.254:38459)
0928 21:17:27.271179 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:27.271183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:27.271319 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:17:27.271464 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:27.271467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:27.271516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:27.271577 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:27.272033 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.272543 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.273329 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.273547 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.273659 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.276029 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:40325: BlockingRecv error: recv got EOF from 127.5.18.195:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:27 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:27.942799 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:27.931408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:48561)
0928 21:17:27.931664 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:27.931677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:27.931766 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:27.931992 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:27.931994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:27.932158 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:27.932249 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.932254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.932917 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.932919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.933475 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:27.933483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.933887 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.933889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:27.933946 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.940843 (+ 6897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:27.940860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:27.942641 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
W20260928 21:17:27.942947 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:27.931492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:48561 (local address 127.5.18.254:38459)
0928 21:17:27.931745 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:27.931748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:27.931760 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:27.931842 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:27.931844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:27.931884 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:27.931964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:27.932328 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.932844 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.933582 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:27.933806 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:27.933878 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:27.942808 (+ 8930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:48561: BlockingRecv error: recv got EOF from 127.5.18.193:48561 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:28.101598 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:28.096461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:59931)
0928 21:17:28.096719 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:28.096735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:28.096845 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:28.097068 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:28.097071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:28.097286 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:28.097379 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.097386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.098245 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.098249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.098799 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.098807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.099229 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.099233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.099315 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.099798 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:28.099810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:28.101461 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
W20260928 21:17:28.101743 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:28.096521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:59931 (local address 127.5.18.254:38459)
0928 21:17:28.096683 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:28.096687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:28.096804 (+ 117us) server_negotiation.cc:407] Connection header received
0928 21:17:28.096926 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:28.096929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:28.096977 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:28.097039 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:28.097490 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.098137 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.098908 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.099129 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.099211 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.101612 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:59931: BlockingRecv error: recv got EOF from 127.5.18.194:59931 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:28.281520 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:28.276726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:39297)
0928 21:17:28.276976 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:28.276988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:28.277082 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:28.277315 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:28.277317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:28.277481 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:28.277597 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.277603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.278292 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.278294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.278835 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.278841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.279264 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.279266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.279324 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.279786 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:28.279797 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:28.281406 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20260928 21:17:28.281685 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:28.276763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:39297 (local address 127.5.18.254:38459)
0928 21:17:28.276892 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:28.276903 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:28.277050 (+ 147us) server_negotiation.cc:407] Connection header received
0928 21:17:28.277148 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:28.277151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:28.277224 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:28.277296 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:28.277676 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.278211 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.278933 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.279156 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.279264 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.281538 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:39297: BlockingRecv error: recv got EOF from 127.5.18.195:39297 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:28 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:28.951262 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:28.943781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:59077)
0928 21:17:28.945320 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:28.945339 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:28.945474 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:28.945838 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:28.945841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:28.946071 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:28.946197 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.946206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.947280 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.947284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.948100 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:28.948111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.948708 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.948711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:28.948780 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.949294 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:28.949305 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:28.951142 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1450,"thread_start_us":89,"threads_started":1}
W20260928 21:17:28.951406 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:28.943812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:59077 (local address 127.5.18.254:38459)
0928 21:17:28.944165 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:28.944170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:28.945599 (+ 1429us) server_negotiation.cc:407] Connection header received
0928 21:17:28.945648 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:28.945651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:28.945707 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:28.945810 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:28.946327 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.947115 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.948292 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:28.948583 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:28.948670 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:28.951285 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:59077: BlockingRecv error: recv got EOF from 127.5.18.193:59077 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":66,"threads_started":1}
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:29.108263 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:29.102268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:43439)
0928 21:17:29.102571 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:29.102584 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:29.102682 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:29.102926 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:29.102928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:29.103087 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:29.103179 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.103184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.104199 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.104201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.104740 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.104747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.105356 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.105361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.105451 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.106165 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:29.106186 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:29.108151 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
W20260928 21:17:29.108572 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:29.102384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43439 (local address 127.5.18.254:38459)
0928 21:17:29.102537 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:29.102543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:29.102646 (+ 103us) server_negotiation.cc:407] Connection header received
0928 21:17:29.102752 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:29.102755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:29.102812 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:29.102903 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:29.103275 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.104107 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.104834 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.105211 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.105325 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.108318 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:43439: BlockingRecv error: recv got EOF from 127.5.18.194:43439 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:29.287213 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:29.282228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:43147)
0928 21:17:29.282509 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:29.282524 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:29.282642 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:29.282880 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:29.282883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:29.283070 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:29.283158 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.283164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.283880 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.283884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.284460 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.284469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.284883 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.284887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.284964 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.285408 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:29.285419 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:29.287068 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
W20260928 21:17:29.287308 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:29.282325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:43147 (local address 127.5.18.254:38459)
0928 21:17:29.282486 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:29.282490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:29.282590 (+ 100us) server_negotiation.cc:407] Connection header received
0928 21:17:29.282725 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:29.282728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:29.282777 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:29.282844 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:29.283268 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.283780 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.284572 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.284789 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.284873 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.287200 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:43147: BlockingRecv error: recv got EOF from 127.5.18.195:43147 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:29 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:17:29.957864 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:29.952102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:55241 (local address 127.5.18.254:38459)
0928 21:17:29.952344 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:29.952348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:29.952482 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:17:29.952618 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:29.952621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:29.952668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:29.952742 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:29.953196 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.953802 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.954576 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.954798 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.954881 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.957759 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:55241: BlockingRecv error: recv got EOF from 127.5.18.193:55241 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":50,"threads_started":1}
I20260928 21:17:29.957855 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:29.952029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:55241)
0928 21:17:29.952381 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:29.952415 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:29.952534 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:29.952784 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:29.952787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:29.952981 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:29.953085 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.953092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.953899 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.953903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.954468 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:29.954476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:29.954894 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:29.954898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:29.954980 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:29.955497 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:29.955517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:29.957659 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:30.114454 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:30.109172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:54811)
0928 21:17:30.109496 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:30.109510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:30.109631 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:30.109853 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:30.109856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:30.110084 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:30.110200 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.110207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.110925 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.110929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.111491 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.111498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.111914 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.111917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.112012 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.112498 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:30.112508 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:30.114331 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":77,"threads_started":1}
W20260928 21:17:30.114624 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:30.109299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:54811 (local address 127.5.18.254:38459)
0928 21:17:30.109466 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:30.109470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:30.109582 (+ 112us) server_negotiation.cc:407] Connection header received
0928 21:17:30.109713 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:30.109716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:30.109764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:30.109825 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:30.110304 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.110827 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.111593 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.111819 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.111904 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.114509 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:54811: BlockingRecv error: recv got EOF from 127.5.18.194:54811 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:30.293372 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:30.288057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:51969)
0928 21:17:30.288323 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:30.288335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:30.288438 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:30.288671 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:30.288674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:30.288865 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:30.288966 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.288972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.289754 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.289759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.290510 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.290525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.291009 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.291012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.291082 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.291539 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:30.291551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:30.293225 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20260928 21:17:30.293453 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:30.288137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:51969 (local address 127.5.18.254:38459)
0928 21:17:30.288268 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:30.288273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:30.288401 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:17:30.288525 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:30.288528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:30.288575 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:30.288638 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:30.289079 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.289624 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.290620 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.290892 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.291001 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.293357 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:51969: BlockingRecv error: recv got EOF from 127.5.18.195:51969 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:30 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:30.963852 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:30.958589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:48949)
0928 21:17:30.958842 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:30.958855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:30.958945 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:30.959155 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:30.959157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:30.959321 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:30.959438 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.959443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.960343 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.960348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.961056 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:30.961068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.961498 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.961501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:30.961616 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.962076 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:30.962088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:30.963726 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
W20260928 21:17:30.964015 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:30.958642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:48949 (local address 127.5.18.254:38459)
0928 21:17:30.958878 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:30.958881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:30.958898 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:17:30.959017 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:30.959019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:30.959061 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:30.959132 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:30.959531 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.960202 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.961171 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:30.961387 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:30.961471 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:30.963871 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:48949: BlockingRecv error: recv got EOF from 127.5.18.193:48949 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:31.120384 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:31.115184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:55769)
0928 21:17:31.115439 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:31.115453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:31.115568 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:31.115793 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:31.115795 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:31.116008 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:31.116112 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.116118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.116930 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.116934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.117493 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.117502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.117926 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.117930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.118017 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.118524 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:31.118535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:31.120249 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
W20260928 21:17:31.120468 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:31.115280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:55769 (local address 127.5.18.254:38459)
0928 21:17:31.115439 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:31.115443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:31.115522 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:17:31.115652 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:31.115655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:31.115700 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:31.115762 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:31.116223 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.116827 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.117611 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.117829 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.117911 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.120375 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:55769: BlockingRecv error: recv got EOF from 127.5.18.194:55769 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:31.299718 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:31.294108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:59975)
0928 21:17:31.294371 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:31.294385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:31.294616 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:31.294956 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:31.294959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:31.295171 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:31.295268 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.295274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.296068 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.296072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.296639 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.296648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.297091 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.297096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.297185 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.297708 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:31.297721 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:31.299561 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
W20260928 21:17:31.299800 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:31.294199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:59975 (local address 127.5.18.254:38459)
0928 21:17:31.294363 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:31.294368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:31.294578 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:17:31.294720 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:31.294724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:31.294776 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:31.294917 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:31.295420 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.295953 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.296758 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.296984 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.297080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.299686 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:59975: BlockingRecv error: recv got EOF from 127.5.18.195:59975 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:31 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:31.970037 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:31.964706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:48083)
0928 21:17:31.964994 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:31.965020 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:31.965112 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:31.965331 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:31.965333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:31.965506 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:31.965630 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.965635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.966391 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.966393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.966943 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:31.966951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.967451 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.967454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:31.967521 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.967987 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:31.967997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:31.969888 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":69,"threads_started":1}
W20260928 21:17:31.970180 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:31.964803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:48083 (local address 127.5.18.254:38459)
0928 21:17:31.965042 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:31.965046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:31.965062 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:17:31.965187 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:31.965189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:31.965231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:31.965307 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:31.965713 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.966312 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.967056 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:31.967343 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:31.967458 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:31.970034 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:48083: BlockingRecv error: recv got EOF from 127.5.18.193:48083 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:32.126318 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:32.121060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:39409)
0928 21:17:32.121416 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:32.121429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:32.121542 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:32.121798 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:32.121801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:32.122003 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:32.122096 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.122102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.122814 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.122818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.123378 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.123386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.123792 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.123796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.123874 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.124390 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:32.124402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:32.126203 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
W20260928 21:17:32.126477 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:32.121139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:39409 (local address 127.5.18.254:38459)
0928 21:17:32.121314 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:32.121318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:32.121498 (+ 180us) server_negotiation.cc:407] Connection header received
0928 21:17:32.121646 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:32.121649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:32.121696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:32.121758 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:32.122205 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.122715 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.123483 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.123699 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.123780 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.126354 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:39409: BlockingRecv error: recv got EOF from 127.5.18.194:39409 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:32.305688 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:32.300504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:42995)
0928 21:17:32.300792 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:32.300804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:32.300911 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:32.301122 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:32.301125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:32.301298 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:32.301389 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.301394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.302188 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.302192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.302948 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.302958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.303381 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.303384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.303467 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.303901 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:32.303912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:32.305506 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":78,"threads_started":1}
W20260928 21:17:32.305770 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:32.300572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:42995 (local address 127.5.18.254:38459)
0928 21:17:32.300684 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:32.300688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:32.300866 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:17:32.300978 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:32.300980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:32.301038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:32.301099 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:32.301467 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.302095 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.303062 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.303285 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.303367 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.305672 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:42995: BlockingRecv error: recv got EOF from 127.5.18.195:42995 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:32 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:32.981866 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:32.970727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:37113)
0928 21:17:32.970987 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:32.971003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:32.971103 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:32.971300 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:32.971301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:32.971473 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:32.971575 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.971580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.972280 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.972282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.972854 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:32.972863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.973289 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.973292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:32.973354 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.979832 (+ 6478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:32.979844 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:32.981729 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
W20260928 21:17:32.982038 14372 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:32.970884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:37113 (local address 127.5.18.254:38459)
0928 21:17:32.971112 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:32.971115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:32.971128 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:32.971161 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:32.971163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:32.971203 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:32.971274 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:32.971656 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.972185 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.972968 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:32.973200 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:32.973275 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:32.981892 (+ 8617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:37113: BlockingRecv error: recv got EOF from 127.5.18.193:37113 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:33.132206 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:33.127091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:35983)
0928 21:17:33.127348 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:33.127360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:33.127481 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:33.127672 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:33.127674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:33.127847 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:33.127943 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.127948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.128788 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.128791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.129362 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.129371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.129769 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.129772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.129831 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.130261 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:33.130271 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:33.132076 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":78,"threads_started":1}
W20260928 21:17:33.132332 14372 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:33.127134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:35983 (local address 127.5.18.254:38459)
0928 21:17:33.127336 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:33.127340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:33.127453 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:17:33.127559 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:33.127561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:33.127603 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:33.127658 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:33.128026 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.128688 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.129465 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.129690 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.129757 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.132227 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:35983: BlockingRecv error: recv got EOF from 127.5.18.194:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:33.311277 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:33.306471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:58767)
0928 21:17:33.306727 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:33.306743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:33.306855 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:33.307131 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:33.307134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:33.307351 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:33.307473 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.307479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.308163 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.308165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.308710 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.308716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.309078 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.309080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.309134 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.309624 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:33.309638 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:33.311165 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
W20260928 21:17:33.311379 14372 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:33.306515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:58767 (local address 127.5.18.254:38459)
0928 21:17:33.306648 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:33.306652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:33.306817 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:17:33.306955 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:33.306958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:33.307014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:33.307089 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:33.307574 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.308082 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.308790 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.309001 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.309069 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.311282 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:58767: BlockingRecv error: recv got EOF from 127.5.18.195:58767 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:33 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:33.987824 14375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:33.982639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:50561)
0928 21:17:33.982954 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:33.982967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:33.983082 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:33.983317 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:33.983318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:33.983485 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:33.983586 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.983593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.984430 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.984433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.985122 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:33.985131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.985533 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.985535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:33.985630 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.986071 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:33.986082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:33.987711 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
W20260928 21:17:33.987949 14376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:33.982725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:50561 (local address 127.5.18.254:38459)
0928 21:17:33.982983 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:33.982986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:33.983033 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:17:33.983162 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:33.983164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:33.983211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:33.983294 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:33.983706 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.984322 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.985217 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:33.985447 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:33.985525 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:33.987835 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:50561: BlockingRecv error: recv got EOF from 127.5.18.193:50561 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":63,"threads_started":1}
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:34.137764 14377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:34.132909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:48369)
0928 21:17:34.133190 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:34.133205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:34.133304 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:34.133516 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:34.133518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:34.133706 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:34.133799 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.133805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.134462 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.134464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.135002 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.135008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.135379 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.135381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.135443 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.135909 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:34.135919 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:34.137632 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
W20260928 21:17:34.137825 14376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:34.132971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:48369 (local address 127.5.18.254:38459)
0928 21:17:34.133147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:34.133152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:34.133259 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:17:34.133378 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:34.133380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:34.133433 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:34.133489 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:34.133882 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.134388 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.135082 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.135297 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.135363 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.137734 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:48369: BlockingRecv error: recv got EOF from 127.5.18.194:48369 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:34.317276 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:34.311976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:43433)
0928 21:17:34.312247 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:34.312260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:34.312355 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:34.312558 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:34.312561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:34.312739 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:34.312832 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.312837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.313555 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.313559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.314300 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.314311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.314752 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.314755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.314837 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.315326 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:34.315339 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:34.317070 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
W20260928 21:17:34.317303 14376 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:34.312054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:43433 (local address 127.5.18.254:38459)
0928 21:17:34.312196 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:34.312201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:34.312311 (+ 110us) server_negotiation.cc:407] Connection header received
0928 21:17:34.312432 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:34.312434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:34.312479 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:34.312536 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:34.312914 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.313419 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.314407 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.314651 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.314738 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.317204 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:43433: BlockingRecv error: recv got EOF from 127.5.18.195:43433 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:34 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:34.993680 14379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:34.988535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:40471)
0928 21:17:34.988833 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:34.988847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:34.988948 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:34.989185 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:34.989187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:34.989379 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:34.989469 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.989474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.990235 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.990238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.990798 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:34.990806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.991230 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.991234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:34.991314 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.991806 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:34.991818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:34.993496 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":70,"threads_started":1}
W20260928 21:17:34.993775 14380 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:34.988584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:40471 (local address 127.5.18.254:38459)
0928 21:17:34.988808 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:34.988812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:34.988906 (+ 94us) server_negotiation.cc:407] Connection header received
0928 21:17:34.989036 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:34.989039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:34.989085 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:34.989155 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:34.989608 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.990135 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.990906 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:34.991135 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:34.991218 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:34.993660 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:40471: BlockingRecv error: recv got EOF from 127.5.18.193:40471 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:35.143601 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:35.138459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:41291)
0928 21:17:35.138719 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:35.138731 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:35.138826 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:35.139029 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:35.139031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:35.139215 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:35.139311 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.139315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.140102 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.140105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.140671 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.140679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.141088 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.141090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.141158 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.141687 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:35.141699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:35.143435 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
W20260928 21:17:35.143682 14380 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:35.138527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:41291 (local address 127.5.18.254:38459)
0928 21:17:35.138668 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:35.138672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:35.138793 (+ 121us) server_negotiation.cc:407] Connection header received
0928 21:17:35.138894 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:35.138897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:35.138941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:35.139012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:35.139460 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.139998 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.140784 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.141001 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.141083 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.143581 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:41291: BlockingRecv error: recv got EOF from 127.5.18.194:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:35.323129 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:35.318129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:50745)
0928 21:17:35.318394 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:35.318407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:35.318493 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:35.318707 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:35.318710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:35.318891 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:35.319008 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.319015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.319835 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.319838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.320386 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.320394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.320778 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.320780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.320836 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.321274 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:35.321285 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:35.323004 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
W20260928 21:17:35.323251 14380 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:35.318265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:50745 (local address 127.5.18.254:38459)
0928 21:17:35.318395 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:35.318399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:35.318467 (+ 68us) server_negotiation.cc:407] Connection header received
0928 21:17:35.318574 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:35.318576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:35.318620 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:35.318685 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:35.319175 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.319741 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.320480 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.320697 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.320771 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.323125 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:50745: BlockingRecv error: recv got EOF from 127.5.18.195:50745 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:35 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:35.999491 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:35.994481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:36301)
0928 21:17:35.994749 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:35.994762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:35.994847 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:35.995063 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:35.995065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:35.995236 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:35.995332 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.995336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.996004 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.996006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.996559 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:35.996566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.996928 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.996931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:35.996991 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.997439 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:35.997450 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:35.999344 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
W20260928 21:17:35.999577 14384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:35.994562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:36301 (local address 127.5.18.254:38459)
0928 21:17:35.994788 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:35.994791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:35.994804 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:35.994918 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:35.994920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:35.994962 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:35.995034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:35.995412 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.995931 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.996635 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:35.996848 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:35.996912 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:35.999465 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:36301: BlockingRecv error: recv got EOF from 127.5.18.193:36301 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:36.149698 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:36.144362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:32911)
0928 21:17:36.144616 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:36.144629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:36.144729 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:36.144986 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:36.144988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:36.145161 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:36.145256 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:36.145262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.145981 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.145986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:36.146764 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:36.146777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.147262 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.147265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:36.147337 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:36.147822 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:36.147833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:36.149538 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
W20260928 21:17:36.149775 14384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:36.144442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:32911 (local address 127.5.18.254:38459)
0928 21:17:36.144607 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:36.144610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:36.144697 (+ 87us) server_negotiation.cc:407] Connection header received
0928 21:17:36.144835 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:36.144837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:36.144888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:36.144951 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:36.145343 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.145875 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.146893 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.147134 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.147227 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:36.149682 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:32911: BlockingRecv error: recv got EOF from 127.5.18.194:32911 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:36.328779 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:36.323807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:40057)
0928 21:17:36.324041 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:36.324054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:36.324147 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:36.324350 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:36.324353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:36.324518 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:36.324610 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:36.324616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.325273 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.325275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:36.325858 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:36.325866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.326260 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.326264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:36.326344 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:36.326875 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:36.326890 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:36.328663 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
W20260928 21:17:36.328886 14384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:36.323895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:40057 (local address 127.5.18.254:38459)
0928 21:17:36.324014 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:36.324018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:36.324110 (+ 92us) server_negotiation.cc:407] Connection header received
0928 21:17:36.324215 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:36.324217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:36.324261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:36.324329 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:36.324693 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.325196 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.325939 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:36.326160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:36.326232 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:36.328788 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:40057: BlockingRecv error: recv got EOF from 127.5.18.195:40057 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:36 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:37.005511 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:37.000343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:47437)
0928 21:17:37.000625 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:37.000640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:37.000741 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:37.000961 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:37.000963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:37.001160 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:37.001263 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.001269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.002001 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.002003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.002568 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.002577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.002960 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.002963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.003038 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.003524 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:37.003538 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:37.005362 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
W20260928 21:17:37.005674 14388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:37.000431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:47437 (local address 127.5.18.254:38459)
0928 21:17:37.000692 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:37.000695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:37.000707 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:37.000813 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:37.000815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:37.000858 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:37.000930 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:37.001354 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.001917 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.002661 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.002875 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.002941 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.005492 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:47437: BlockingRecv error: recv got EOF from 127.5.18.193:47437 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
Sep 28 21:17:37 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:37 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:37.155941 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:37.150421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:52491)
0928 21:17:37.150701 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:37.150716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:37.150832 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:37.151059 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:37.151062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:37.151260 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:37.151351 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.151357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.152182 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.152186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.152807 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.152817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.153268 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.153272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.153352 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.153888 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:37.153900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:37.155798 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
W20260928 21:17:37.156028 14388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:37.150541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:52491 (local address 127.5.18.254:38459)
0928 21:17:37.150703 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:37.150707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:37.150788 (+ 81us) server_negotiation.cc:407] Connection header received
0928 21:17:37.150915 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:37.150918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:37.150964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:37.151028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:37.151457 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.152081 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.152915 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.153170 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.153253 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.155930 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:52491: BlockingRecv error: recv got EOF from 127.5.18.194:52491 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 21:17:37 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:37 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:37.335041 14390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:37.329540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:49085)
0928 21:17:37.329831 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:37.329846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:37.329948 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:37.330166 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:37.330168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:37.330343 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:37.330438 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.330443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.331171 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.331174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.331755 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:37.331766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.332229 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.332232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:37.332314 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.333004 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:37.333018 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:37.334907 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20260928 21:17:37.335187 14388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:37.329634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:49085 (local address 127.5.18.254:38459)
0928 21:17:37.329773 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:37.329776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:37.329908 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:17:37.330016 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:37.330018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:37.330070 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:37.330143 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:37.330525 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.331034 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.331889 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:37.332121 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:37.332210 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:37.335062 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:49085: BlockingRecv error: recv got EOF from 127.5.18.195:49085 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:38.014323 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:38.006361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:33661)
0928 21:17:38.006605 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:38.006618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:38.006711 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:38.006929 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:38.006931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:38.007101 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:38.007194 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.007198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.007865 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.007868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.008406 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.008413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.008804 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.008807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.008880 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.012469 (+ 3589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:38.012481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:38.014200 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
W20260928 21:17:38.014487 14392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:38.006420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:33661 (local address 127.5.18.254:38459)
0928 21:17:38.006646 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:38.006650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:38.006667 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:17:38.006789 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:38.006791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:38.006833 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:38.006907 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:38.007273 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.007787 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.008483 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.008691 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.008757 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.014356 (+ 5599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:33661: BlockingRecv error: recv got EOF from 127.5.18.193:33661 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:38.161723 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:38.156584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:50417)
0928 21:17:38.156839 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:38.156855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:38.156966 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:38.157202 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:38.157205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:38.157401 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:38.157492 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.157498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.158267 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.158270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.158839 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.158847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.159271 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.159274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.159352 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.159844 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:38.159857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:38.161548 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
W20260928 21:17:38.161984 14392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:38.156719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:50417 (local address 127.5.18.254:38459)
0928 21:17:38.156874 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:38.156879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:38.156914 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:17:38.157053 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:38.157055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:38.157102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:38.157167 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:38.157647 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.158163 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.158949 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.159172 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.159258 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.161718 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:50417: BlockingRecv error: recv got EOF from 127.5.18.194:50417 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:38 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:38.341223 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:38.335760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:43295)
0928 21:17:38.336028 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:38.336045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:38.336179 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:38.336431 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:38.336434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:38.336649 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:38.336748 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.336755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.337526 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.337530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.338258 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:38.338269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.338734 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.338739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:38.338821 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.339371 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:38.339382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:38.341096 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
W20260928 21:17:38.341333 14392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:38.335863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:43295 (local address 127.5.18.254:38459)
0928 21:17:38.336017 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:38.336023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:38.336127 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:17:38.336266 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:38.336270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:38.336325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:38.336398 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:38.336880 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.337414 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.338383 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:38.338623 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:38.338723 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:38.341223 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:43295: BlockingRecv error: recv got EOF from 127.5.18.195:43295 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:39.019986 14395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:39.015078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:58079)
0928 21:17:39.015329 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:39.015341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:39.015428 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:39.015628 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:39.015630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:39.015829 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:39.015919 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.015925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.016626 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.016628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.017188 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.017195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.017599 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.017603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.017670 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.018160 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:39.018171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:39.019870 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
W20260928 21:17:39.020112 14396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:39.015160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:58079 (local address 127.5.18.254:38459)
0928 21:17:39.015426 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:39.015430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:39.015441 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:17:39.015495 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:39.015497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:39.015549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:39.015619 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:39.016014 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.016548 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.017270 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.017493 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.017591 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.019986 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:58079: BlockingRecv error: recv got EOF from 127.5.18.193:58079 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":55,"threads_started":1}
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:39.167454 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:39.162437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:59155)
0928 21:17:39.162696 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:39.162709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:39.162803 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:39.163001 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:39.163003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:39.163176 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:39.163268 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.163273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.163963 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.163966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.164573 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.164583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.164942 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.164944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.165002 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.165465 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:39.165479 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:39.167294 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
W20260928 21:17:39.167834 14396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:39.162520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:59155 (local address 127.5.18.254:38459)
0928 21:17:39.162643 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:39.162647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:39.162756 (+ 109us) server_negotiation.cc:407] Connection header received
0928 21:17:39.162877 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:39.162879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:39.162921 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:39.162976 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:39.163348 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.163849 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.164665 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.164866 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.164931 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.167726 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:59155: BlockingRecv error: recv got EOF from 127.5.18.194:59155 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:39 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.195@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:39.346843 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:39.341915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.195:50837)
0928 21:17:39.342219 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:39.342233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:39.342328 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:39.342535 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:39.342537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:39.342711 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:39.342805 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.342810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.343504 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.343507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.344081 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:39.344089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.344465 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.344468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:39.344526 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.344970 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:39.344981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:39.346699 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
W20260928 21:17:39.346937 14396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:39.342000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:50837 (local address 127.5.18.254:38459)
0928 21:17:39.342127 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:39.342131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:39.342283 (+ 152us) server_negotiation.cc:407] Connection header received
0928 21:17:39.342406 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:39.342408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:39.342454 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:39.342508 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:39.342899 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.343410 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.344167 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:39.344381 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:39.344454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:39.346836 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.195:50837: BlockingRecv error: recv got EOF from 127.5.18.195:50837 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 21:17:40 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:40 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.193@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:40.026037 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.020850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.193:56215)
0928 21:17:40.021133 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:40.021163 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:40.021270 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:40.021600 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:40.021604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:40.021809 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:40.021916 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.021923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.022623 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.022626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.023205 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.023214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.023583 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.023585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.023641 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.024099 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:40.024115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:40.025921 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
W20260928 21:17:40.026155 14400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:40.020895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56215 (local address 127.5.18.254:38459)
0928 21:17:40.021127 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:40.021131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:40.021233 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:17:40.021365 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:40.021368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:40.021421 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:40.021495 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:40.022014 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.022533 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.023288 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.023509 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.023574 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.026045 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.193:56215: BlockingRecv error: recv got EOF from 127.5.18.193:56215 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":58,"threads_started":1}
Sep 28 21:17:40 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:17:40 dist-test-slave-rc9f krb5kdc[13341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.18.194@KRBTEST.COM for oryx/127.5.18.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:17:40.172895 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.168259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38459 (local address 127.5.18.194:42569)
0928 21:17:40.168495 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:40.168510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:40.168589 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:40.168804 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:40.168806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:40.168974 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:40.169066 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.169071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.169747 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.169749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.170322 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.170329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.170690 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.170693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.170749 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.171180 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:40.171191 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:40.172775 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.18.254:38459: Server oryx/127.5.18.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
W20260928 21:17:40.172992 14400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:40.168424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:42569 (local address 127.5.18.254:38459)
0928 21:17:40.168539 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:40.168542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:40.168555 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:40.168661 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:40.168663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:40.168711 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:40.168777 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:40.169145 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.169673 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.170414 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.170617 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.170684 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.172897 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.18.194:42569: BlockingRecv error: recv got EOF from 127.5.18.194:42569 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:17:40.257369 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13924
I20260928 21:17:40.262158 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14075
I20260928 21:17:40.266577 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14209
I20260928 21:17:40.270777 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 13854
2026-09-28T21:17:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17736 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:40Z Disabled control of system clock
I20260928 21:17:40.297911 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:41335
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:41335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:17:40.404107 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:40.404346 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:40.404386 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:40.408336 14405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:40.408388 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:40.408401 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:40.408416 14405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:40.408432 14405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:40.413172 14405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:41335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:41335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:40.414183 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:40.415494 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:40.423661 14410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:40.423755 14411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:40.423668 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:40.423731 14405 server_base.cc:1034] running on GCE node
I20260928 21:17:40.424477 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:40.425451 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:40.426604 14405 hybrid_clock.cc:648] HybridClock initialized: now 1790630260426583 us; error 36 us; skew 500 ppm
I20260928 21:17:40.428359 14405 webserver.cc:460] Webserver started at http://127.5.18.254:43483/ using document root <none> and password file <none>
I20260928 21:17:40.428869 14405 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:40.428925 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:40.429091 14405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:40.431020 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "49c739bd5cac4f40ba65316c5ed98966"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "54096d5b284793f64e14078e31dcdb6e"
server_key_iv: "5dd7095e2a2087ba28ce7af7a199c249"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.431486 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "49c739bd5cac4f40ba65316c5ed98966"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "54096d5b284793f64e14078e31dcdb6e"
server_key_iv: "5dd7095e2a2087ba28ce7af7a199c249"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.434887 14405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:17:40.437068 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.438417 14405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:17:40.438541 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "49c739bd5cac4f40ba65316c5ed98966"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "54096d5b284793f64e14078e31dcdb6e"
server_key_iv: "5dd7095e2a2087ba28ce7af7a199c249"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.438638 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:40.450348 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:40.450946 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:40.451099 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:40.457351 14405 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:41335
I20260928 21:17:40.457371 14471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:41335 every 8 connection(s)
I20260928 21:17:40.458264 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:40.460938 14472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:40.462965 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14405
I20260928 21:17:40.463083 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:40.463367 5195 external_mini_cluster.cc:1428] Setting key 7e234771026db9dc643e2da41bf6f144
I20260928 21:17:40.466822 14472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966: Bootstrap starting.
I20260928 21:17:40.468880 14472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:40.469537 14472 log.cc:824] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:40.471330 14472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966: No bootstrap required, opened a new log
I20260928 21:17:40.471956 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.464740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.5.18.254:41335)
0928 21:17:40.465343 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:40.465375 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:40.465409 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:40.466050 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:40.466067 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:40.466463 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:40.466755 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:40.467158 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.468079 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.468921 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.469408 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.469546 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.470366 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:40.470371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:40.470386 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:40.470740 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:40.470792 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:40.470819 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:40.470900 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:40.471034 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 21:17:40.471266 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260928 21:17:40.474666 14472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } }
I20260928 21:17:40.474864 14472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:40.474924 14472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c739bd5cac4f40ba65316c5ed98966, State: Initialized, Role: FOLLOWER
I20260928 21:17:40.475281 14472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } }
I20260928 21:17:40.475411 14472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:40.475471 14472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:40.475564 14472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:40.476327 14472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } }
I20260928 21:17:40.476634 14472 leader_election.cc:302] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c739bd5cac4f40ba65316c5ed98966; no voters:
I20260928 21:17:40.476882 14472 leader_election.cc:288] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:40.476958 14477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:40.477149 14477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [term 1 LEADER]: Becoming Leader. State: Replica: 49c739bd5cac4f40ba65316c5ed98966, State: Running, Role: LEADER
I20260928 21:17:40.477427 14477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } }
I20260928 21:17:40.477959 14472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:40.478907 14479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c739bd5cac4f40ba65316c5ed98966. Latest consensus state: current_term: 1 leader_uuid: "49c739bd5cac4f40ba65316c5ed98966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } } }
I20260928 21:17:40.479027 14479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:40.479300 14478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c739bd5cac4f40ba65316c5ed98966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c739bd5cac4f40ba65316c5ed98966" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 41335 } } }
I20260928 21:17:40.479456 14478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:40.479480 14486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:40.482465 14486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:40.487561 14486 catalog_manager.cc:1347] Generated new cluster ID: cec8baac2b81451fb6dfc0b05d1e7d33
I20260928 21:17:40.487613 14486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:40.503216 14486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:40.504324 14486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:40.510316 14486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c739bd5cac4f40ba65316c5ed98966: Generated new TSK 0
I20260928 21:17:40.511211 14486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:40.523283 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--builtin_ntp_servers=127.5.18.212:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:17:40.631042 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:40.631286 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:40.631323 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:40.635316 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:40.635380 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:40.635437 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:40.640251 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:40.641405 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:40.642894 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:40.652410 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:40.652539 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:40.652565 14496 server_base.cc:1034] running on GCE node
W20260928 21:17:40.652443 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:40.653009 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:40.653729 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:40.654899 14496 hybrid_clock.cc:648] HybridClock initialized: now 1790630260654872 us; error 40 us; skew 500 ppm
I20260928 21:17:40.656735 14496 webserver.cc:460] Webserver started at http://127.5.18.193:39613/ using document root <none> and password file <none>
I20260928 21:17:40.657300 14496 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:40.657363 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:40.657594 14496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:40.659700 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "2d416fb8818443a7b4864ed920327c6a"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "14edde412d73174c8d36238b52d41182"
server_key_iv: "007760b516820c3d121f7a423f23d1a2"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.660264 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "2d416fb8818443a7b4864ed920327c6a"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "14edde412d73174c8d36238b52d41182"
server_key_iv: "007760b516820c3d121f7a423f23d1a2"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.664088 14496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 21:17:40.666685 14510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.668020 14496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:40.668138 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "2d416fb8818443a7b4864ed920327c6a"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "14edde412d73174c8d36238b52d41182"
server_key_iv: "007760b516820c3d121f7a423f23d1a2"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.668262 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:40.686869 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:40.687547 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:40.687719 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:40.688294 14496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:40.689218 14496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:40.689262 14496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.689306 14496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:40.689332 14496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.697310 14496 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:44349
I20260928 21:17:40.697360 14623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:44349 every 8 connection(s)
I20260928 21:17:40.698238 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:40.699208 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14496
I20260928 21:17:40.699312 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:40.699620 5195 external_mini_cluster.cc:1428] Setting key 3ec7f46b07593d66a71c09a178fe3ba8
I20260928 21:17:40.701620 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--builtin_ntp_servers=127.5.18.212:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:17:40.708496 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.700153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:53923 (local address 127.5.18.254:41335)
0928 21:17:40.700302 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:40.700307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:40.701481 (+ 1174us) server_negotiation.cc:407] Connection header received
0928 21:17:40.703047 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:40.703050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:40.703101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:40.703170 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:40.704227 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.704770 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.706526 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.706759 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.706856 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.708050 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:40.708053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:40.708055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:40.708067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:40.708086 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:40.708094 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:40.708151 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:40.708327 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 21:17:40.708381 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:17:40.709709 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.700679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:41335 (local address 127.5.18.193:53923)
0928 21:17:40.701354 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:40.701395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:40.702832 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:40.703349 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:40.703357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:40.703445 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:40.704077 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.704096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.704904 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.704908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.706332 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.706344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.706916 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.706921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.707125 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.707741 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:40.707744 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:40.707802 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:40.707887 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:40.708171 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:40.708179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:40.708185 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:17:40.708316 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 21:17:40.708633 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260928 21:17:40.710841 14624 heartbeater.cc:347] Connected to a master server at 127.5.18.254:41335
I20260928 21:17:40.711098 14624 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:40.711714 14624 heartbeater.cc:511] Master 127.5.18.254:41335 requested a full tablet report, sending...
I20260928 21:17:40.713126 14436 ts_manager.cc:194] Registered new tserver with Master: 2d416fb8818443a7b4864ed920327c6a (127.5.18.193:44349)
I20260928 21:17:40.714264 14436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.193:53923
W20260928 21:17:40.808975 14627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:40.809243 14627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:40.809293 14627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:40.813287 14627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:40.813349 14627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:40.813422 14627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:40.818095 14627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:40.819124 14627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:40.820431 14627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:40.829427 14632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:40.829522 14635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:40.829546 14627 server_base.cc:1034] running on GCE node
W20260928 21:17:40.829409 14633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:40.830123 14627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:40.830711 14627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:40.831867 14627 hybrid_clock.cc:648] HybridClock initialized: now 1790630260831846 us; error 32 us; skew 500 ppm
I20260928 21:17:40.833596 14627 webserver.cc:460] Webserver started at http://127.5.18.194:38281/ using document root <none> and password file <none>
I20260928 21:17:40.834144 14627 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:40.834203 14627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:40.834396 14627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:40.836341 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "7eb4918e39484442841131a9414f422c"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "d72a8b27f5c61a687775bebeb7b53455"
server_key_iv: "1df1995342f4320f1bce2c3c94ea90ee"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.836862 14627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "7eb4918e39484442841131a9414f422c"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "d72a8b27f5c61a687775bebeb7b53455"
server_key_iv: "1df1995342f4320f1bce2c3c94ea90ee"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.840313 14627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:40.842664 14641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.843786 14627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:40.843887 14627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "7eb4918e39484442841131a9414f422c"
format_stamp: "Formatted at 2026-09-28 21:17:40 on dist-test-slave-rc9f"
server_key: "d72a8b27f5c61a687775bebeb7b53455"
server_key_iv: "1df1995342f4320f1bce2c3c94ea90ee"
server_key_version: "encryptionkey@0"
I20260928 21:17:40.843982 14627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:40.867962 14627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:40.868654 14627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:40.868829 14627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:40.869415 14627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:40.870402 14627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:40.870450 14627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.870491 14627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:40.870517 14627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:40.878732 14627 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37645
I20260928 21:17:40.878764 14754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37645 every 8 connection(s)
I20260928 21:17:40.879673 14627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:40.887630 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14627
I20260928 21:17:40.887729 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:40.887975 5195 external_mini_cluster.cc:1428] Setting key fd00a10ddfec30425d5f94949d9f1e7f
I20260928 21:17:40.888305 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.881394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:48081 (local address 127.5.18.254:41335)
0928 21:17:40.881558 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:40.881578 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:40.882631 (+ 1053us) server_negotiation.cc:407] Connection header received
0928 21:17:40.883940 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:40.883943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:40.883988 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:40.884061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:40.884930 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.885453 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.886561 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.886790 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.886868 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.887852 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:40.887855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:40.887857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:40.887870 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:40.887889 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:40.887897 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:40.887955 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:40.888100 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:17:40.888162 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:17:40.889158 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:40.881863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:41335 (local address 127.5.18.194:48081)
0928 21:17:40.882493 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:40.882534 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:40.883732 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:40.884213 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:40.884220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:40.884306 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:40.884805 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.884817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.885539 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.885541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.886450 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:40.886459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:40.886905 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:40.886907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:40.887039 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:40.887596 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:40.887598 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:40.887637 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:40.887705 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:40.887964 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:40.887973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:40.887979 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:17:40.888090 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:17:40.888376 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260928 21:17:40.889894 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--builtin_ntp_servers=127.5.18.212:42393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:17:40.890262 14755 heartbeater.cc:347] Connected to a master server at 127.5.18.254:41335
I20260928 21:17:40.890501 14755 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:40.891081 14755 heartbeater.cc:511] Master 127.5.18.254:41335 requested a full tablet report, sending...
I20260928 21:17:40.892086 14436 ts_manager.cc:194] Registered new tserver with Master: 7eb4918e39484442841131a9414f422c (127.5.18.194:37645)
I20260928 21:17:40.892632 14436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.194:48081
W20260928 21:17:41.002746 14758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:41.002969 14758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:41.003000 14758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:41.006861 14758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:41.006906 14758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:41.006953 14758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:41.011484 14758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:42393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:41335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:41.012370 14758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:41.013602 14758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:41.022107 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.022117 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.022125 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.022150 14758 server_base.cc:1034] running on GCE node
I20260928 21:17:41.022697 14758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:41.023249 14758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:41.024468 14758 hybrid_clock.cc:648] HybridClock initialized: now 1790630261024448 us; error 31 us; skew 500 ppm
I20260928 21:17:41.026330 14758 webserver.cc:460] Webserver started at http://127.5.18.195:40177/ using document root <none> and password file <none>
I20260928 21:17:41.026845 14758 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:41.026906 14758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:41.027077 14758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:41.028988 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "54b2e954aac8434daef0173d80d651c8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "32ad4293bfac55cfd8753c852af18453"
server_key_iv: "793622cd693c830bb2db953e21f1ceb9"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.029505 14758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "54b2e954aac8434daef0173d80d651c8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "32ad4293bfac55cfd8753c852af18453"
server_key_iv: "793622cd693c830bb2db953e21f1ceb9"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.032804 14758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:17:41.035151 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.036305 14758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:41.036394 14758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "54b2e954aac8434daef0173d80d651c8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "32ad4293bfac55cfd8753c852af18453"
server_key_iv: "793622cd693c830bb2db953e21f1ceb9"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.036496 14758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:41.064211 14758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:41.064877 14758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:41.065034 14758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:41.065621 14758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:41.066547 14758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:41.066604 14758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.066654 14758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:41.066682 14758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.074522 14758 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:33213
I20260928 21:17:41.074553 14885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:33213 every 8 connection(s)
I20260928 21:17:41.075357 14758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:41.076284 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14758
I20260928 21:17:41.076385 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:41.076653 5195 external_mini_cluster.cc:1428] Setting key 188768b995867fe5f25f16af00dbae79
I20260928 21:17:41.083794 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.077147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:40215 (local address 127.5.18.254:41335)
0928 21:17:41.077277 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.077281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.078435 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 21:17:41.079790 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.079793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.079863 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.079924 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.080722 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:41.081236 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:41.082320 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:41.082531 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:41.082604 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:41.083455 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:41.083458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:41.083460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:41.083470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:41.083487 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:41.083494 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:41.083544 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:41.083641 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:17:41.083685 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:17:41.084543 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.077549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:41335 (local address 127.5.18.195:40215)
0928 21:17:41.078297 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:41.078348 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:41.079616 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:41.080063 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:41.080070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:41.080150 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:41.080615 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:41.080628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:41.081317 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:41.081320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:41.082217 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:41.082234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:41.082632 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:41.082634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:41.082762 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:41.083249 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:41.083251 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:41.083287 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:41.083344 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:41.083550 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:41.083555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:41.083558 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:41.083635 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:17:41.083828 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260928 21:17:41.085454 14886 heartbeater.cc:347] Connected to a master server at 127.5.18.254:41335
I20260928 21:17:41.085667 14886 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:41.086129 14886 heartbeater.cc:511] Master 127.5.18.254:41335 requested a full tablet report, sending...
I20260928 21:17:41.086997 14436 ts_manager.cc:194] Registered new tserver with Master: 54b2e954aac8434daef0173d80d651c8 (127.5.18.195:33213)
I20260928 21:17:41.087627 14436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.195:40215
I20260928 21:17:41.090265 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:17:41.092891 14475 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:41.091986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.5.18.254:41335)
0928 21:17:41.092103 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.092108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.092274 (+ 166us) server_negotiation.cc:407] Connection header received
0928 21:17:41.092411 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.092414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.092456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.092518 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.092777 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41008: BlockingRecv error: recv got EOF from 127.0.0.1:41008 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:17:41.093094 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14496
I20260928 21:17:41.098244 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14627
I20260928 21:17:41.102655 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14758
I20260928 21:17:41.107004 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14405
2026-09-28T21:17:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (834 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:17:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:41Z Disabled control of system clock
I20260928 21:17:41.133049 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38733
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:17:41.238281 14897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:41.238538 14897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:41.238575 14897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:41.242528 14897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:41.242583 14897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:41.242601 14897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:41.242621 14897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:41.242662 14897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:41.247354 14897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:41.248334 14897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:41.249713 14897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:41.257355 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.257361 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.257361 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.257514 14897 server_base.cc:1034] running on GCE node
I20260928 21:17:41.258155 14897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:41.259176 14897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:41.260324 14897 hybrid_clock.cc:648] HybridClock initialized: now 1790630261260310 us; error 25 us; skew 500 ppm
I20260928 21:17:41.262006 14897 webserver.cc:460] Webserver started at http://127.5.18.254:41757/ using document root <none> and password file <none>
I20260928 21:17:41.262503 14897 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:41.262565 14897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:41.262741 14897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:41.264626 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "006fbd74e5e14eacbc7cff49bb70b071"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "83be61c12028f3a8cfc6b46f54d2b833"
server_key_iv: "e80ae458fd62e99dd1e2b570309ec1bd"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.265141 14897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "006fbd74e5e14eacbc7cff49bb70b071"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "83be61c12028f3a8cfc6b46f54d2b833"
server_key_iv: "e80ae458fd62e99dd1e2b570309ec1bd"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.268491 14897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:17:41.270680 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.271814 14897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:41.271929 14897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "006fbd74e5e14eacbc7cff49bb70b071"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "83be61c12028f3a8cfc6b46f54d2b833"
server_key_iv: "e80ae458fd62e99dd1e2b570309ec1bd"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.272017 14897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:41.276311 14897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:41.276872 14897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:41.277020 14897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:41.283313 14897 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38733
I20260928 21:17:41.283330 14963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38733 every 8 connection(s)
I20260928 21:17:41.284166 14897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:41.287117 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:41.287883 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14897
I20260928 21:17:41.287979 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:41.288223 5195 external_mini_cluster.cc:1428] Setting key a9944beb0a02d982e5ec9e457ef89219
I20260928 21:17:41.293349 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071: Bootstrap starting.
I20260928 21:17:41.294006 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.289521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.5.18.254:38733)
0928 21:17:41.290202 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.290221 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.290260 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:17:41.291058 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.291080 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.291506 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.291870 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.292526 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:41.292531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:41.292545 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:41.292555 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:41.292614 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:41.292646 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:41.292777 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:41.292850 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:17:41.293083 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
I20260928 21:17:41.295733 14964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:41.296428 14964 log.cc:824] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:41.298105 14964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071: No bootstrap required, opened a new log
I20260928 21:17:41.301352 14964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } }
I20260928 21:17:41.301553 14964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:41.301622 14964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006fbd74e5e14eacbc7cff49bb70b071, State: Initialized, Role: FOLLOWER
I20260928 21:17:41.301971 14964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } }
I20260928 21:17:41.302075 14964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:41.302117 14964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:41.302181 14964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:41.302943 14964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } }
I20260928 21:17:41.303212 14964 leader_election.cc:302] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006fbd74e5e14eacbc7cff49bb70b071; no voters:
I20260928 21:17:41.303453 14964 leader_election.cc:288] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:41.303542 14969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:41.303695 14969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [term 1 LEADER]: Becoming Leader. State: Replica: 006fbd74e5e14eacbc7cff49bb70b071, State: Running, Role: LEADER
I20260928 21:17:41.303978 14969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } }
I20260928 21:17:41.304657 14964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:41.305612 14971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006fbd74e5e14eacbc7cff49bb70b071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } } }
I20260928 21:17:41.305657 14970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006fbd74e5e14eacbc7cff49bb70b071. Latest consensus state: current_term: 1 leader_uuid: "006fbd74e5e14eacbc7cff49bb70b071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006fbd74e5e14eacbc7cff49bb70b071" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38733 } } }
I20260928 21:17:41.305734 14971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:41.305757 14970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:41.306108 14978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:41.308995 14978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:41.313939 14978 catalog_manager.cc:1347] Generated new cluster ID: 91711b9b390442649421446323b63ba4
I20260928 21:17:41.313998 14978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:41.325241 14978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:41.325549 14978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:41.332067 14978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006fbd74e5e14eacbc7cff49bb70b071: Generated new TSK 0
I20260928 21:17:41.332855 14978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:17:41.336520 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--builtin_ntp_servers=127.5.18.212:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:17:41.441761 14988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:41.442006 14988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:41.442055 14988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:41.446012 14988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:41.446067 14988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:41.446151 14988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:41.450824 14988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:41.451853 14988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:41.453207 14988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:41.462246 14994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.462296 14988 server_base.cc:1034] running on GCE node
W20260928 21:17:41.462211 14996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.462249 14993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.462770 14988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:41.463332 14988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:41.464488 14988 hybrid_clock.cc:648] HybridClock initialized: now 1790630261464464 us; error 35 us; skew 500 ppm
I20260928 21:17:41.466197 14988 webserver.cc:460] Webserver started at http://127.5.18.193:36421/ using document root <none> and password file <none>
I20260928 21:17:41.466713 14988 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:41.466770 14988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:41.466956 14988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:41.468863 14988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "2ce59661bfcd495883b9d591c413abfa"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "f1519d43efae64b28b7e7e80373fbd24"
server_key_iv: "684254c3fb9621cf9287152d29faa4ac"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.469379 14988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "2ce59661bfcd495883b9d591c413abfa"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "f1519d43efae64b28b7e7e80373fbd24"
server_key_iv: "684254c3fb9621cf9287152d29faa4ac"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.472797 14988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:17:41.475152 15002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.476361 14988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:41.476456 14988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "2ce59661bfcd495883b9d591c413abfa"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "f1519d43efae64b28b7e7e80373fbd24"
server_key_iv: "684254c3fb9621cf9287152d29faa4ac"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.476545 14988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:41.480700 14988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:41.481252 14988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:41.481401 14988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:41.481953 14988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:41.482770 14988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:41.482815 14988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.482861 14988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:41.482889 14988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.490505 14988 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:41759
I20260928 21:17:41.490569 15115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:41759 every 8 connection(s)
I20260928 21:17:41.491285 14988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:41.491895 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 14988
I20260928 21:17:41.491984 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:41.492233 5195 external_mini_cluster.cc:1428] Setting key db7bb769c5844e98a15454aa1d15970e
I20260928 21:17:41.494258 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--builtin_ntp_servers=127.5.18.212:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:17:41.497543 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.493009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:59739 (local address 127.5.18.254:38733)
0928 21:17:41.493136 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.493141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.494360 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 21:17:41.495961 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.495964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.496006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.496076 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.497149 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:41.497152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:41.497154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:41.497167 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:41.497189 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:41.497199 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:41.497278 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:41.497406 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 21:17:41.497444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:17:41.498674 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.493414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38733 (local address 127.5.18.193:59739)
0928 21:17:41.494217 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:41.494268 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:41.495760 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:41.496231 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:41.496238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:41.496323 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:41.496857 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:41.496859 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:41.496910 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:41.496990 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:41.497276 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:41.497283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:41.497288 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:17:41.497447 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 21:17:41.497710 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":175,"threads_started":1}
I20260928 21:17:41.499902 15116 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38733
I20260928 21:17:41.500099 15116 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:41.500468 15116 heartbeater.cc:511] Master 127.5.18.254:38733 requested a full tablet report, sending...
I20260928 21:17:41.501929 14928 ts_manager.cc:194] Registered new tserver with Master: 2ce59661bfcd495883b9d591c413abfa (127.5.18.193:41759)
W20260928 21:17:41.600190 15119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:41.600443 15119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:41.600495 15119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:41.604429 15119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:41.604481 15119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:41.604553 15119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:41.609194 15119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:41.610204 15119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:41.611496 15119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:41.620363 15125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.620349 15124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.620352 15127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.620615 15119 server_base.cc:1034] running on GCE node
I20260928 21:17:41.621014 15119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:41.621623 15119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:41.622785 15119 hybrid_clock.cc:648] HybridClock initialized: now 1790630261622754 us; error 44 us; skew 500 ppm
I20260928 21:17:41.624529 15119 webserver.cc:460] Webserver started at http://127.5.18.194:37687/ using document root <none> and password file <none>
I20260928 21:17:41.625123 15119 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:41.625285 15119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:41.625540 15119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:41.627815 15119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "5107e7811613430faf2f221ce37863f8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "5a21e8bce7da3a195b22cb4955c558e7"
server_key_iv: "fe275bf70fca08d3014745002091217b"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.628387 15119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "5107e7811613430faf2f221ce37863f8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "5a21e8bce7da3a195b22cb4955c558e7"
server_key_iv: "fe275bf70fca08d3014745002091217b"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.632174 15119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:17:41.634653 15133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.635993 15119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:41.636111 15119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "5107e7811613430faf2f221ce37863f8"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "5a21e8bce7da3a195b22cb4955c558e7"
server_key_iv: "fe275bf70fca08d3014745002091217b"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.636215 15119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:41.640625 15119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:41.641196 15119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:41.641346 15119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:41.641908 15119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:41.642910 15119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:41.642966 15119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.643020 15119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:41.643059 15119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.650990 15119 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:35111
I20260928 21:17:41.651038 15246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:35111 every 8 connection(s)
I20260928 21:17:41.651866 15119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:41.657406 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.653522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:33079 (local address 127.5.18.254:38733)
0928 21:17:41.653686 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.653691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.654782 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 21:17:41.656009 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.656012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.656049 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.656121 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.657076 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:41.657079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:41.657081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:41.657092 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:41.657110 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:41.657119 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:41.657203 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:41.657268 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:17:41.657303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:17:41.658205 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.653974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38733 (local address 127.5.18.194:33079)
0928 21:17:41.654640 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:41.654679 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:41.655809 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:41.656263 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:41.656269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:41.656363 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:41.656841 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:41.656843 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:41.656880 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:41.656945 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:41.657173 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:41.657177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:41.657180 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:41.657290 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:17:41.657478 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260928 21:17:41.659219 15247 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38733
I20260928 21:17:41.659422 15247 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:41.659781 15247 heartbeater.cc:511] Master 127.5.18.254:38733 requested a full tablet report, sending...
I20260928 21:17:41.660089 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15119
I20260928 21:17:41.660190 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:41.660470 5195 external_mini_cluster.cc:1428] Setting key 700bc296cdf010337108e1637fef72cd
I20260928 21:17:41.660718 14928 ts_manager.cc:194] Registered new tserver with Master: 5107e7811613430faf2f221ce37863f8 (127.5.18.194:35111)
I20260928 21:17:41.662358 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--builtin_ntp_servers=127.5.18.212:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:17:41.767374 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:41.767602 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:41.767642 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:41.771574 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:41.771620 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:41.771679 15250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:41.776232 15250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:38733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:41.777161 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:41.778481 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:41.787528 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.787529 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:41.787516 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:41.787827 15250 server_base.cc:1034] running on GCE node
I20260928 21:17:41.788188 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:41.788774 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:41.789916 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790630261789894 us; error 38 us; skew 500 ppm
I20260928 21:17:41.791518 15250 webserver.cc:460] Webserver started at http://127.5.18.195:35853/ using document root <none> and password file <none>
I20260928 21:17:41.792028 15250 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:41.792090 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:41.792259 15250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:41.794304 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "49922fcefb0848d8b03767f5404c0e8a"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "8ae01c8a3fe6028b78c00ee1546ae69f"
server_key_iv: "98f654db0ad9b03856417d2034f3c96f"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.794816 15250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "49922fcefb0848d8b03767f5404c0e8a"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "8ae01c8a3fe6028b78c00ee1546ae69f"
server_key_iv: "98f654db0ad9b03856417d2034f3c96f"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.798179 15250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:17:41.800417 15264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.801512 15250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:41.801635 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "49922fcefb0848d8b03767f5404c0e8a"
format_stamp: "Formatted at 2026-09-28 21:17:41 on dist-test-slave-rc9f"
server_key: "8ae01c8a3fe6028b78c00ee1546ae69f"
server_key_iv: "98f654db0ad9b03856417d2034f3c96f"
server_key_version: "encryptionkey@0"
I20260928 21:17:41.801725 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:41.805935 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:41.806500 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:41.806643 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:41.807135 15250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:41.807953 15250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:41.807997 15250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.808043 15250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:41.808070 15250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:41.816028 15250 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:45291
I20260928 21:17:41.816090 15377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:45291 every 8 connection(s)
I20260928 21:17:41.816813 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:41.817495 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15250
I20260928 21:17:41.817612 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:41.817930 5195 external_mini_cluster.cc:1428] Setting key a0ca36a015cc28a152ea24cb7e40ccb5
I20260928 21:17:41.822408 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.818559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:44879 (local address 127.5.18.254:38733)
0928 21:17:41.818684 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.818688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.819896 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 21:17:41.821077 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.821079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.821111 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.821165 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.822110 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:41.822113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:41.822115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:17:41.822123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:41.822138 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:17:41.822144 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:41.822203 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:41.822278 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:17:41.822309 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:17:41.823184 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:41.818988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38733 (local address 127.5.18.195:44879)
0928 21:17:41.819742 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:41.819793 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:41.820909 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:41.821294 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:41.821301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:41.821381 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:41.821906 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:17:41.821909 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:41.821946 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:17:41.822009 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:41.822194 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:41.822197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:41.822200 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:17:41.822306 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:17:41.822485 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20260928 21:17:41.824188 15378 heartbeater.cc:347] Connected to a master server at 127.5.18.254:38733
I20260928 21:17:41.824381 15378 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:41.824698 15378 heartbeater.cc:511] Master 127.5.18.254:38733 requested a full tablet report, sending...
I20260928 21:17:41.825619 14928 ts_manager.cc:194] Registered new tserver with Master: 49922fcefb0848d8b03767f5404c0e8a (127.5.18.195:45291)
I20260928 21:17:41.831367 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:17:41.833949 14967 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:17:41.833080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.5.18.254:38733)
0928 21:17:41.833200 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:41.833205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:41.833362 (+ 157us) server_negotiation.cc:407] Connection header received
0928 21:17:41.833500 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:41.833503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:41.833539 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:41.833631 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:41.833825 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58354: BlockingRecv error: recv got EOF from 127.0.0.1:58354 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:17:41.834074 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14988
I20260928 21:17:41.839644 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15119
I20260928 21:17:41.843977 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15250
I20260928 21:17:41.848276 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 14897
2026-09-28T21:17:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (741 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:41 dist-test-slave-rc9f krb5kdc[15388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:41 dist-test-slave-rc9f krb5kdc[15388](info): set up 2 sockets
Sep 28 21:17:41 dist-test-slave-rc9f krb5kdc[15388](info): commencing operation
krb5kdc: starting...
W20260928 21:17:42.798048 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:42 dist-test-slave-rc9f krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:17:42.907894 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45195
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:45195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:43.013006 15401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:43.013253 15401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:43.013305 15401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:43.017226 15401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:17:43.017316 15401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:43.017364 15401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:43.017418 15401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:17:43.017450 15401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:17:43.022612 15401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:45195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:43.023564 15401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:43.024508 15401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:43.032217 15406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.032356 15407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.032420 15401 server_base.cc:1034] running on GCE node
W20260928 21:17:43.032322 15409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.033036 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:43.034070 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:43.035225 15401 hybrid_clock.cc:648] HybridClock initialized: now 1790630263035207 us; error 30 us; skew 500 ppm
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:43.043780 15401 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:17:43.044564 15401 webserver.cc:460] Webserver started at http://127.5.18.254:45591/ using document root <none> and password file <none>
I20260928 21:17:43.045047 15401 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:43.045099 15401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:43.045258 15401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:43.047185 15401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "3973f53995a0438eb7484bc7c8006843"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "90441df087d574013f4a96e062d37d92"
server_key_iv: "1ed085813b01f26dc4d854eb4c6e672f"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.047652 15401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "3973f53995a0438eb7484bc7c8006843"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "90441df087d574013f4a96e062d37d92"
server_key_iv: "1ed085813b01f26dc4d854eb4c6e672f"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.051306 15401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 21:17:43.053423 15416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.054571 15401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:43.054663 15401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "3973f53995a0438eb7484bc7c8006843"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "90441df087d574013f4a96e062d37d92"
server_key_iv: "1ed085813b01f26dc4d854eb4c6e672f"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.054764 15401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:43.066069 15401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:43.068776 15401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:43.068904 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:43.075069 15401 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:45195
I20260928 21:17:43.075297 15468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:45195 every 8 connection(s)
I20260928 21:17:43.075937 15401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:17:43.078483 15469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:43.082952 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15401
I20260928 21:17:43.083076 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:17:43.083365 5195 external_mini_cluster.cc:1428] Setting key ba6e37daadff5e2b1560bcca48f957b8
I20260928 21:17:43.083945 15469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: Bootstrap starting.
I20260928 21:17:43.086190 15469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:43.086860 15469 log.cc:824] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:43.088488 15469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: No bootstrap required, opened a new log
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:43.091928 15469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } }
I20260928 21:17:43.092123 15469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:43.092157 15469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3973f53995a0438eb7484bc7c8006843, State: Initialized, Role: FOLLOWER
I20260928 21:17:43.092576 15469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } }
I20260928 21:17:43.092693 15469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:17:43.092748 15469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:17:43.092835 15469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:43.093590 15469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } }
I20260928 21:17:43.093886 15469 leader_election.cc:302] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3973f53995a0438eb7484bc7c8006843; no voters:
I20260928 21:17:43.094141 15469 leader_election.cc:288] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:17:43.094213 15474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:43.094408 15474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [term 1 LEADER]: Becoming Leader. State: Replica: 3973f53995a0438eb7484bc7c8006843, State: Running, Role: LEADER
I20260928 21:17:43.094735 15474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } }
I20260928 21:17:43.095340 15469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:17:43.096917 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.084653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.5.18.254:45195)
0928 21:17:43.085037 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.085050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.085084 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:17:43.085715 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.085732 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.086060 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.086352 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:43.087103 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.088042 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.088954 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.089431 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.089595 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.092028 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:43.092033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:43.092045 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:43.092069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:43.093734 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.094129 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.094135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.094141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.094286 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.094534 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.094537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.094539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.094999 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:43.095182 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.095719 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.096014 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260928 21:17:43.097767 15476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3973f53995a0438eb7484bc7c8006843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } } }
I20260928 21:17:43.097891 15475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3973f53995a0438eb7484bc7c8006843. Latest consensus state: current_term: 1 leader_uuid: "3973f53995a0438eb7484bc7c8006843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3973f53995a0438eb7484bc7c8006843" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45195 } } }
I20260928 21:17:43.098254 15475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [sys.catalog]: This master's current role is: LEADER
I20260928 21:17:43.098251 15476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843 [sys.catalog]: This master's current role is: LEADER
W20260928 21:17:43.100244 15489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:17:43.100315 15489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:17:43.100373 15490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:17:43.101061 15490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:17:43.105903 15490 catalog_manager.cc:1347] Generated new cluster ID: 692633751d3945bdb27edcbc6adcacd8
I20260928 21:17:43.105954 15490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:17:43.114733 15490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:17:43.115814 15490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:17:43.127307 15490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: Generated new TSK 0
I20260928 21:17:43.128127 15490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:17:43.153764 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:43.255934 15495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:43.256152 15495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:43.256191 15495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:43.260145 15495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:43.260193 15495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:43.260257 15495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:17:43.265134 15495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:43.266203 15495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:43.267200 15495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:43.276404 15503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.276381 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.276384 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.276432 15495 server_base.cc:1034] running on GCE node
I20260928 21:17:43.276947 15495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:43.277511 15495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:43.278692 15495 hybrid_clock.cc:648] HybridClock initialized: now 1790630263278677 us; error 32 us; skew 500 ppm
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:43.281271 15495 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:17:43.282155 15495 webserver.cc:460] Webserver started at http://127.5.18.193:37641/ using document root <none> and password file <none>
I20260928 21:17:43.282675 15495 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:43.282774 15495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:43.282971 15495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:43.284911 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "d12fed402c864a28ad8719fb624d5c8f"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "fee6a971dc3d3025d518c256b334dd67"
server_key_iv: "48d2cadaff2c6bff2276c242af128347"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.285421 15495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "d12fed402c864a28ad8719fb624d5c8f"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "fee6a971dc3d3025d518c256b334dd67"
server_key_iv: "48d2cadaff2c6bff2276c242af128347"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.289198 15495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:17:43.291535 15510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.292688 15495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:17:43.292806 15495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "d12fed402c864a28ad8719fb624d5c8f"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "fee6a971dc3d3025d518c256b334dd67"
server_key_iv: "48d2cadaff2c6bff2276c242af128347"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.292903 15495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:43.312361 15495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:43.315299 15495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:43.315459 15495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:43.315994 15495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:43.316725 15495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:43.316776 15495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.316825 15495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:43.316854 15495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.324657 15495 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:34773
I20260928 21:17:43.324685 15623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:34773 every 8 connection(s)
I20260928 21:17:43.325433 15495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:17:43.328992 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15495
I20260928 21:17:43.329087 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:17:43.329366 5195 external_mini_cluster.cc:1428] Setting key d4cc835bf6171a0fff32e87c991ef74d
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:17:43.338521 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.327156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:34201 (local address 127.5.18.254:45195)
0928 21:17:43.327318 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.327323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.328067 (+ 744us) server_negotiation.cc:407] Connection header received
0928 21:17:43.329362 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.329366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.329423 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.329513 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:43.330901 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.331626 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.332775 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.333036 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.333126 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.335368 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:43.335370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:43.335372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:43.335394 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:43.336652 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.337098 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.337102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.337103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.337170 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.337460 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.337464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.337465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.337667 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:43.337751 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.338204 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.338328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:17:43.339221 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.327498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45195 (local address 127.5.18.193:34201)
0928 21:17:43.327926 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.327966 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.329154 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.329692 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.329700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.330137 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:43.330752 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.330773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.331749 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.331752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.332650 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.332659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.333167 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.333171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.333334 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.333880 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:43.333895 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:43.335220 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:43.336763 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.336769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.336782 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.337009 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.337247 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.337250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.337252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.337379 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.337769 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:43.337776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:43.337994 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.338150 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.338439 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:17:43.340355 15624 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45195
I20260928 21:17:43.340590 15624 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:43.341116 15624 heartbeater.cc:511] Master 127.5.18.254:45195 requested a full tablet report, sending...
I20260928 21:17:43.342542 15433 ts_manager.cc:194] Registered new tserver with Master: d12fed402c864a28ad8719fb624d5c8f (127.5.18.193:34773)
I20260928 21:17:43.343641 15433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:34201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:17:43.354447 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:43.459753 15629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:43.460002 15629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:43.460065 15629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:43.464004 15629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:43.464063 15629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:43.464154 15629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:17:43.469130 15629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:43.470151 15629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:43.471134 15629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:17:43.480450 15629 server_base.cc:1034] running on GCE node
W20260928 21:17:43.480366 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.480427 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.480366 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.480976 15629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:43.481597 15629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:43.482772 15629 hybrid_clock.cc:648] HybridClock initialized: now 1790630263482757 us; error 28 us; skew 500 ppm
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:43.485282 15629 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:17:43.486135 15629 webserver.cc:460] Webserver started at http://127.5.18.194:34699/ using document root <none> and password file <none>
I20260928 21:17:43.486640 15629 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:43.486707 15629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:43.486894 15629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:43.488794 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "943532731cac4938ab6ee902302dbcb6"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "94aa04397698a81e814f3a7689b1b915"
server_key_iv: "f6268c29ed8855540fba8c805a6a74ff"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.489336 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "943532731cac4938ab6ee902302dbcb6"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "94aa04397698a81e814f3a7689b1b915"
server_key_iv: "f6268c29ed8855540fba8c805a6a74ff"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.492842 15629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:17:43.495112 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.496320 15629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:17:43.496424 15629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "943532731cac4938ab6ee902302dbcb6"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "94aa04397698a81e814f3a7689b1b915"
server_key_iv: "f6268c29ed8855540fba8c805a6a74ff"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.496517 15629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:43.519328 15629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:43.522014 15629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:43.522197 15629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:43.522753 15629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:43.523566 15629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:43.523617 15629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.523674 15629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:43.523705 15629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.532547 15629 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43787
I20260928 21:17:43.532569 15757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43787 every 8 connection(s)
I20260928 21:17:43.533464 15629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:17:43.539597 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15629
I20260928 21:17:43.539705 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:17:43.540016 5195 external_mini_cluster.cc:1428] Setting key be802e135cb28234ab65105ca39b933f
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:43.546808 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.535152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:38887 (local address 127.5.18.254:45195)
0928 21:17:43.535311 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.535317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.536114 (+ 797us) server_negotiation.cc:407] Connection header received
0928 21:17:43.537365 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.537368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.537418 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.537498 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:43.538683 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.539244 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.540383 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.540644 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.540756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.543247 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:43.543250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:43.543252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:43.543276 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:43.544633 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.545202 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.545204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.545206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.545299 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.545698 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.545700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.545701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.545885 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:43.545968 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.546571 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.546675 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:17:43.547807 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.535548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45195 (local address 127.5.18.194:38887)
0928 21:17:43.535965 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.536006 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.537167 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.537670 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.537676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.538019 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:43.538538 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.538554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.539341 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.539344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.540265 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.540274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.540794 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.540798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.540995 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.541625 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:43.541643 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:43.543077 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:43.544758 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.544765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.544780 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.545112 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.545391 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.545394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.545396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.545613 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.545985 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:43.545992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:43.546289 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.546488 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.546815 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:17:43.548959 15758 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45195
I20260928 21:17:43.549158 15758 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:43.549647 15758 heartbeater.cc:511] Master 127.5.18.254:45195 requested a full tablet report, sending...
I20260928 21:17:43.550665 15433 ts_manager.cc:194] Registered new tserver with Master: 943532731cac4938ab6ee902302dbcb6 (127.5.18.194:43787)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:17:43.551354 15433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:38887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:17:43.564709 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:39871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:17:43.670492 15763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:17:43.670746 15763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:17:43.670789 15763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:17:43.674793 15763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:17:43.674841 15763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:17:43.674896 15763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:17:43.680014 15763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:39871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:17:43.680917 15763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:17:43.681859 15763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:17:43.691414 15768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:17:43.691402 15769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.691550 15763 server_base.cc:1034] running on GCE node
W20260928 21:17:43.691707 15771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:17:43.692075 15763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:17:43.692663 15763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:17:43.693809 15763 hybrid_clock.cc:648] HybridClock initialized: now 1790630263693778 us; error 45 us; skew 500 ppm
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:17:43.696430 15763 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:17:43.697327 15763 webserver.cc:460] Webserver started at http://127.5.18.195:45255/ using document root <none> and password file <none>
I20260928 21:17:43.697921 15763 fs_manager.cc:359] Metadata directory not provided
I20260928 21:17:43.697980 15763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:17:43.698168 15763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:17:43.700141 15763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "a446805e50e04995be8ada6cac60c7cc"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "9ec483a3fb66e345cc39b5844ccd3d10"
server_key_iv: "3780b573c303632c118613473e583a51"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.700659 15763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "a446805e50e04995be8ada6cac60c7cc"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "9ec483a3fb66e345cc39b5844ccd3d10"
server_key_iv: "3780b573c303632c118613473e583a51"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.703997 15763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:17:43.706238 15778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.707427 15763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:17:43.707520 15763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "a446805e50e04995be8ada6cac60c7cc"
format_stamp: "Formatted at 2026-09-28 21:17:43 on dist-test-slave-rc9f"
server_key: "9ec483a3fb66e345cc39b5844ccd3d10"
server_key_iv: "3780b573c303632c118613473e583a51"
server_key_version: "encryptionkey@0"
I20260928 21:17:43.707621 15763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:17:43.724797 15763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:17:43.727474 15763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:17:43.727638 15763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:17:43.728176 15763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:17:43.728942 15763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:17:43.728987 15763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.729028 15763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:17:43.729058 15763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:17:43.737035 15763 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:36073
I20260928 21:17:43.737064 15891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:36073 every 8 connection(s)
I20260928 21:17:43.737877 15763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:17:43.740151 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 15763
I20260928 21:17:43.740252 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:17:43.740545 5195 external_mini_cluster.cc:1428] Setting key b4eea989d14cc96fe6139fae66e7173a
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:17:43.750392 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.739533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:40535 (local address 127.5.18.254:45195)
0928 21:17:43.739649 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.739652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.740472 (+ 820us) server_negotiation.cc:407] Connection header received
0928 21:17:43.741953 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.741957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.742012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.742078 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:43.743242 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.743761 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.744843 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.745057 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.745136 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.747427 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:43.747430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:43.747432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:43.747457 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:43.748575 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.749022 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.749025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.749027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.749104 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.749398 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.749401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.749403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.749599 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:43.749685 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.750125 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.750240 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:17:43.751113 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.739850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45195 (local address 127.5.18.195:40535)
0928 21:17:43.740331 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.740374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.741761 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.742248 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.742256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.742647 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:17:43.743103 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.743117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.743872 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.743876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.744726 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.744735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.745180 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.745184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.745352 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.745854 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:17:43.745869 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:17:43.747262 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:17:43.748675 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.748680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.748693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.748932 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.749181 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:17:43.749185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:17:43.749187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:17:43.749317 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:17:43.749683 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:17:43.749690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:17:43.749916 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.750072 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.750370 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260928 21:17:43.752067 15892 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45195
I20260928 21:17:43.752274 15892 heartbeater.cc:464] Registering TS with master...
I20260928 21:17:43.752736 15892 heartbeater.cc:511] Master 127.5.18.254:45195 requested a full tablet report, sending...
I20260928 21:17:43.753751 15433 ts_manager.cc:194] Registered new tserver with Master: a446805e50e04995be8ada6cac60c7cc (127.5.18.195:36073)
I20260928 21:17:43.754299 15433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:40535
I20260928 21:17:43.765359 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:17:43.773419 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.767063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.5.18.254:45195)
0928 21:17:43.767197 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.767201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.767322 (+ 121us) server_negotiation.cc:407] Connection header received
0928 21:17:43.767416 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.767418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.767458 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.767524 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:17:43.768087 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.768575 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.769355 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.769588 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.769669 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.770823 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:17:43.770826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:17:43.770828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:17:43.770849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:17:43.771940 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.772273 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.772275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.772277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.772342 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:17:43.772686 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:17:43.772687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:17:43.772688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:17:43.772892 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:17:43.772987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.773192 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.773276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:17:43.775916 15433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:17:43.778864 15433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:17:43.792532 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.787252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.5.18.195:36073)
0928 21:17:43.788016 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.788026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.788044 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:17:43.788103 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.788107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.788241 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.788375 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:43.788736 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.789483 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791089 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.791847 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791950 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.792086 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.792264 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.792374 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":85,"threads_started":1}
I20260928 21:17:43.792529 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.786765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43787 (local address 127.0.0.1:36214)
0928 21:17:43.787240 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.787251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.787321 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.788365 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.788371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.788408 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:43.788602 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.788610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.789618 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.789621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.790925 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.790934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.792146 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.792149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.792228 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.792249 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.792313 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.792374 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":104,"spinlock_wait_cycles":6528,"thread_start_us":121,"threads_started":1}
I20260928 21:17:43.792531 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.786634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:36073 (local address 127.0.0.1:53858)
0928 21:17:43.787119 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.787166 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.787288 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.788365 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.788371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.788406 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:43.788602 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.788610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.789628 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.789632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.790985 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.790995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.791967 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.792084 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.792122 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.792227 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.792332 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":93,"spinlock_wait_cycles":11904,"thread_start_us":62,"threads_started":1}
I20260928 21:17:43.792945 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.787605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.5.18.194:43787)
0928 21:17:43.787881 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.787892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.787927 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:17:43.788003 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.788008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.788211 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.788375 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:43.788718 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.789483 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791031 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.791850 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791950 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.792086 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.792706 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.792814 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
I20260928 21:17:43.793841 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.787016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34773 (local address 127.0.0.1:56618)
0928 21:17:43.788611 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.788621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.788708 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.790042 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.790046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.790065 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:43.790174 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.790181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.791444 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.791447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.792267 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.792275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.793446 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.793450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.793535 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.793555 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.793662 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.793726 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260928 21:17:43.794971 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.787383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.5.18.193:34773)
0928 21:17:43.787660 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.787669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.789628 (+ 1959us) server_negotiation.cc:407] Connection header received
0928 21:17:43.789709 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.789714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.789898 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.790055 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:43.790265 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.791338 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.792384 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.793331 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.793441 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.793589 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.793687 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.793770 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":103,"threads_started":1}
I20260928 21:17:43.796115 15826 tablet_service.cc:1467] Processing CreateTablet for tablet ab3838ee9f4241f1ad09932a11ac793a (DEFAULT_TABLE table=test-table [id=74aee089b7304652b4cdc71a828819db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:43.797132 15826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3838ee9f4241f1ad09932a11ac793a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:43.797302 15558 tablet_service.cc:1467] Processing CreateTablet for tablet ab3838ee9f4241f1ad09932a11ac793a (DEFAULT_TABLE table=test-table [id=74aee089b7304652b4cdc71a828819db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:43.798245 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3838ee9f4241f1ad09932a11ac793a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:43.799544 15692 tablet_service.cc:1467] Processing CreateTablet for tablet ab3838ee9f4241f1ad09932a11ac793a (DEFAULT_TABLE table=test-table [id=74aee089b7304652b4cdc71a828819db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:17:43.800474 15692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab3838ee9f4241f1ad09932a11ac793a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:17:43.803100 15909 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Bootstrap starting.
I20260928 21:17:43.805006 15910 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Bootstrap starting.
I20260928 21:17:43.805176 15909 tablet_bootstrap.cc:654] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:43.806758 15909 log.cc:824] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:43.806865 15911 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Bootstrap starting.
I20260928 21:17:43.807185 15910 tablet_bootstrap.cc:654] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:43.808305 15910 log.cc:824] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:43.809042 15909 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: No bootstrap required, opened a new log
I20260928 21:17:43.809113 15911 tablet_bootstrap.cc:654] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Neither blocks nor log segments found. Creating new log.
I20260928 21:17:43.809238 15909 ts_tablet_manager.cc:1397] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:17:43.809803 15911 log.cc:824] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:17:43.810207 15910 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: No bootstrap required, opened a new log
I20260928 21:17:43.810384 15910 ts_tablet_manager.cc:1397] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:17:43.811309 15911 tablet_bootstrap.cc:492] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: No bootstrap required, opened a new log
I20260928 21:17:43.811614 15911 ts_tablet_manager.cc:1397] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:17:43.813050 15909 raft_consensus.cc:348] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.813304 15909 raft_consensus.cc:374] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:43.813388 15909 raft_consensus.cc:729] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a446805e50e04995be8ada6cac60c7cc, State: Initialized, Role: FOLLOWER
I20260928 21:17:43.813908 15909 consensus_queue.cc:260] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.814039 15910 raft_consensus.cc:348] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.814332 15910 raft_consensus.cc:374] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:43.814544 15892 heartbeater.cc:503] Master 127.5.18.254:45195 was elected leader, sending a full tablet report...
I20260928 21:17:43.814870 15909 ts_tablet_manager.cc:1428] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:17:43.814932 15911 raft_consensus.cc:348] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.815191 15911 raft_consensus.cc:374] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:17:43.815241 15911 raft_consensus.cc:729] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943532731cac4938ab6ee902302dbcb6, State: Initialized, Role: FOLLOWER
I20260928 21:17:43.815366 15910 raft_consensus.cc:729] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12fed402c864a28ad8719fb624d5c8f, State: Initialized, Role: FOLLOWER
I20260928 21:17:43.815773 15910 consensus_queue.cc:260] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.816359 15624 heartbeater.cc:503] Master 127.5.18.254:45195 was elected leader, sending a full tablet report...
I20260928 21:17:43.816589 15910 ts_tablet_manager.cc:1428] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:17:43.817137 15911 consensus_queue.cc:260] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.817966 15758 heartbeater.cc:503] Master 127.5.18.254:45195 was elected leader, sending a full tablet report...
I20260928 21:17:43.818058 15911 ts_tablet_manager.cc:1428] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:17:43.823920 15916 raft_consensus.cc:482] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:17:43.824074 15916 raft_consensus.cc:504] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.824970 15916 leader_election.cc:288] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a446805e50e04995be8ada6cac60c7cc (127.5.18.195:36073), 943532731cac4938ab6ee902302dbcb6 (127.5.18.194:43787)
W20260928 21:17:43.826433 15625 tablet.cc:2369] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:17:43.829013 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.825366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:47019 (local address 127.5.18.194:43787)
0928 21:17:43.825459 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.825463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.825475 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:17:43.825517 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.825520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.825580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.825665 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:43.825949 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.826547 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828001 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.828547 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.828706 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.828839 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.828909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:17:43.829027 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.825155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43787 (local address 127.5.18.193:47019)
0928 21:17:43.825273 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.825290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.825373 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.825667 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.825669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.825684 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:43.825840 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.825847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.826661 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.826664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.827882 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.827892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.828655 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.828732 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.828752 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.828814 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.828889 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":24,"spinlock_wait_cycles":22912}
I20260928 21:17:43.829154 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.825155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:36073 (local address 127.5.18.193:46623)
0928 21:17:43.825623 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.825637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:17:43.825744 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:17:43.825948 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:17:43.825951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:17:43.825966 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:17:43.826068 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.826074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.826927 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.826930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.828067 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:17:43.828074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.828817 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:17:43.828905 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.828926 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:17:43.828986 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:17:43.829042 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":67,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260928 21:17:43.829187 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.825318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:46623 (local address 127.5.18.195:36073)
0928 21:17:43.825433 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.825437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.825707 (+ 270us) server_negotiation.cc:407] Connection header received
0928 21:17:43.825818 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.825821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.825865 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.825926 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:17:43.826154 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.826816 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828159 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.828722 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.828815 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.828857 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.829018 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.829083 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:17:43.829691 15712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3838ee9f4241f1ad09932a11ac793a" candidate_uuid: "d12fed402c864a28ad8719fb624d5c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "943532731cac4938ab6ee902302dbcb6" is_pre_election: true
I20260928 21:17:43.829662 15846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3838ee9f4241f1ad09932a11ac793a" candidate_uuid: "d12fed402c864a28ad8719fb624d5c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a446805e50e04995be8ada6cac60c7cc" is_pre_election: true
I20260928 21:17:43.829934 15846 raft_consensus.cc:2454] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12fed402c864a28ad8719fb624d5c8f in term 0.
I20260928 21:17:43.829936 15712 raft_consensus.cc:2454] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12fed402c864a28ad8719fb624d5c8f in term 0.
I20260928 21:17:43.830307 15513 leader_election.cc:302] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a446805e50e04995be8ada6cac60c7cc, d12fed402c864a28ad8719fb624d5c8f; no voters:
I20260928 21:17:43.830533 15916 raft_consensus.cc:2781] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:17:43.830602 15916 raft_consensus.cc:482] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:17:43.830631 15916 raft_consensus.cc:3037] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:43.831544 15916 raft_consensus.cc:504] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.831833 15916 leader_election.cc:288] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [CANDIDATE]: Term 1 election: Requested vote from peers a446805e50e04995be8ada6cac60c7cc (127.5.18.195:36073), 943532731cac4938ab6ee902302dbcb6 (127.5.18.194:43787)
I20260928 21:17:43.832108 15846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3838ee9f4241f1ad09932a11ac793a" candidate_uuid: "d12fed402c864a28ad8719fb624d5c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a446805e50e04995be8ada6cac60c7cc"
I20260928 21:17:43.832201 15846 raft_consensus.cc:3037] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:43.833104 15846 raft_consensus.cc:2454] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12fed402c864a28ad8719fb624d5c8f in term 1.
I20260928 21:17:43.833316 15712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab3838ee9f4241f1ad09932a11ac793a" candidate_uuid: "d12fed402c864a28ad8719fb624d5c8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "943532731cac4938ab6ee902302dbcb6"
I20260928 21:17:43.833350 15513 leader_election.cc:302] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a446805e50e04995be8ada6cac60c7cc, d12fed402c864a28ad8719fb624d5c8f; no voters:
I20260928 21:17:43.833441 15712 raft_consensus.cc:3037] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:17:43.833483 15916 raft_consensus.cc:2781] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:17:43.833680 15916 raft_consensus.cc:686] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [term 1 LEADER]: Becoming Leader. State: Replica: d12fed402c864a28ad8719fb624d5c8f, State: Running, Role: LEADER
I20260928 21:17:43.834026 15916 consensus_queue.cc:237] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } }
I20260928 21:17:43.834613 15712 raft_consensus.cc:2454] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d12fed402c864a28ad8719fb624d5c8f in term 1.
I20260928 21:17:43.836673 15431 catalog_manager.cc:5495] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f reported cstate change: term changed from 0 to 1, leader changed from <none> to d12fed402c864a28ad8719fb624d5c8f (127.5.18.193). New cstate: current_term: 1 leader_uuid: "d12fed402c864a28ad8719fb624d5c8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d12fed402c864a28ad8719fb624d5c8f" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34773 } health_report { overall_health: HEALTHY } } }
I20260928 21:17:43.857332 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.854058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.5.18.193:34773)
0928 21:17:43.854171 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:17:43.854176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:17:43.854189 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:17:43.854325 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:17:43.854327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:17:43.854381 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:17:43.854465 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:17:43.854684 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.855171 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.855994 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:17:43.856205 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:17:43.856304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:17:43.856559 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:17:43.856969 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:17:43.857072 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:17:43.857141 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 21:17:43.857195 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:17:43.862989 15712 raft_consensus.cc:1261] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 1 FOLLOWER]: Refusing update from remote peer d12fed402c864a28ad8719fb624d5c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:43.862989 15846 raft_consensus.cc:1261] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 1 FOLLOWER]: Refusing update from remote peer d12fed402c864a28ad8719fb624d5c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:17:43.863643 15916 consensus_queue.cc:1035] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "943532731cac4938ab6ee902302dbcb6" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:43.863828 15919 consensus_queue.cc:1035] T ab3838ee9f4241f1ad09932a11ac793a P d12fed402c864a28ad8719fb624d5c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a446805e50e04995be8ada6cac60c7cc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:17:43.871449 15922 mvcc.cc:205] Tried to move back new op lower bound from 7334421560775737344 to 7334421560666464256. Current Snapshot: MvccSnapshot[applied={T|T < 7334421560775737344}]
I20260928 21:17:43.872496 15923 mvcc.cc:205] Tried to move back new op lower bound from 7334421560775737344 to 7334421560666464256. Current Snapshot: MvccSnapshot[applied={T|T < 7334421560775737344}]
I20260928 21:17:43.872669 15924 mvcc.cc:205] Tried to move back new op lower bound from 7334421560775737344 to 7334421560666464256. Current Snapshot: MvccSnapshot[applied={T|T < 7334421560775737344}]
I20260928 21:17:43.879356 15431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:43.879500 15431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:17:43.881669 15431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3973f53995a0438eb7484bc7c8006843: Sending DeleteTablet for 3 replicas of tablet ab3838ee9f4241f1ad09932a11ac793a
I20260928 21:17:43.882347 15826 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3838ee9f4241f1ad09932a11ac793a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53858
I20260928 21:17:43.882383 15692 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3838ee9f4241f1ad09932a11ac793a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:36214
I20260928 21:17:43.882872 15558 tablet_service.cc:1514] Processing DeleteTablet for tablet ab3838ee9f4241f1ad09932a11ac793a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:17:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:56618
I20260928 21:17:43.883118 15932 tablet_replica.cc:330] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: stopping tablet replica
I20260928 21:17:43.883203 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15495
I20260928 21:17:43.884984 15932 raft_consensus.cc:2229] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:43.885315 15932 raft_consensus.cc:2258] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:43.885713 15933 tablet_replica.cc:330] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: stopping tablet replica
I20260928 21:17:43.886004 15933 raft_consensus.cc:2229] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:17:43.886276 15933 raft_consensus.cc:2258] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:17:43.887108 15932 ts_tablet_manager.cc:1905] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:43.887506 15933 ts_tablet_manager.cc:1905] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:17:43.889281 15932 ts_tablet_manager.cc:1918] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:43.889364 15932 log.cc:1196] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/ab3838ee9f4241f1ad09932a11ac793a
I20260928 21:17:43.889673 15932 ts_tablet_manager.cc:1939] T ab3838ee9f4241f1ad09932a11ac793a P a446805e50e04995be8ada6cac60c7cc: Deleting consensus metadata
I20260928 21:17:43.890053 15933 ts_tablet_manager.cc:1918] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:17:43.890139 15933 log.cc:1196] T ab3838ee9f4241f1ad09932a11ac793a P 943532731cac4938ab6ee902302dbcb6: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/ab3838ee9f4241f1ad09932a11ac793a
W20260928 21:17:43.890720 15420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:34773 (error 108)
I20260928 21:17:43.890820 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15629
I20260928 21:17:43.890827 15419 catalog_manager.cc:4841] TS a446805e50e04995be8ada6cac60c7cc (127.5.18.195:36073): tablet ab3838ee9f4241f1ad09932a11ac793a (table test-table [id=74aee089b7304652b4cdc71a828819db]) successfully deleted
I20260928 21:17:43.892076 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.891838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34773 (local address 127.0.0.1:56626)
0928 21:17:43.891950 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.892009 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 21:17:43.892277 15420 catalog_manager.cc:4584] TS d12fed402c864a28ad8719fb624d5c8f (127.5.18.193:34773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab3838ee9f4241f1ad09932a11ac793a: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34773: connect: Connection refused (error 111)
I20260928 21:17:43.896049 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15763
I20260928 21:17:43.897066 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:17:43.896189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43787 (local address 127.0.0.1:36216)
0928 21:17:43.896939 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 21:17:43.896990 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:43787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673}
I20260928 21:17:43.900560 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 15401
2026-09-28T21:17:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2052 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15938](info): set up 2 sockets
Sep 28 21:17:43 dist-test-slave-rc9f krb5kdc[15938](info): commencing operation
krb5kdc: starting...
W20260928 21:17:44.842198 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:17:44 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:17:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangeradmin_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangeradmin_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangerlookup_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangerlookup_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
I20260928 21:17:44.998453 5195 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/postgres -l logfile start
2026-09-28 21:17:47.294 UTC [15968] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:17:47.294 UTC [15968] LOG: listening on IPv4 address "127.5.18.212", port 54555
2026-09-28 21:17:47.297 UTC [15968] LOG: listening on Unix socket "/tmp/.s.PGSQL.54555"
2026-09-28 21:17:47.325 UTC [15971] LOG: database system was shut down at 2026-09-28 21:17:46 UTC
2026-09-28 21:17:47.329 UTC [15968] LOG: database system is ready to accept connections
I20260928 21:17:48.194355 5195 mini_postgres.cc:96] Postgres bound to 54555
2026-09-28 21:17:48.200 UTC [15979] FATAL: database "slave" does not exist
127.5.18.212:54555 - accepting connections
I20260928 21:17:48.200764 5195 mini_ranger.cc:161] Starting Ranger...
I20260928 21:17:48.214250 5195 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:17:48.648900 5195 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:17:48.648972 5195 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin
2026-09-28 21:17:48,924 [I] DB FLAVOR :POSTGRES
2026-09-28 21:17:48,924 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:17:48,924 [I] Checking connection..
2026-09-28 21:17:48,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:17:49,154 [I] Checking connection passed.
2026-09-28 21:17:49,154 [I] --------- Verifying version history table ---------
2026-09-28 21:17:49,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:17:49,405 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:17:49,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:17:49,642 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:17:49,643 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:17:49,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:17:49.872 UTC [16043] WARNING: there is no transaction in progress
2026-09-28 21:17:49,877 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:17:49,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:17:50,120 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:17:50,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:17:50,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:17:50,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:17:50,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rc9f.c.gcp-upstream.internal','N') ;"
2026-09-28 21:17:50,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:17:51,014 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:17:51,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:17:51,272 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:17:51,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:17:51,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:17:51,738 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:17:51,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:17:52.690 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.705 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.721 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.736 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.752 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.767 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.819 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.826 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.839 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.849 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.860 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.873 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.882 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.888 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.894 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:52.900 UTC [16151] WARNING: there is no transaction in progress
2026-09-28 21:17:53,517 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:17:53,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:17:53,738 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:17:53,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:17:54,003 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:17:54,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:17:54,259 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:17:54,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:17:54,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:17:54,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rc9f.c.gcp-upstream.internal';"
2026-09-28 21:17:54,964 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:17:54,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:17:55,209 [I] DB_PATCHES have already been applied
I20260928 21:17:55.215008 5195 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyuKJ5W/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:17:55.215073 5195 mini_ranger.cc:193] Using host: 127.5.18.212
I20260928 21:17:55.218384 5195 mini_ranger.cc:227] Ranger admin URL: http://127.5.18.212:42119
Sep 28, 2026 9:17:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:17:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:17:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:17:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:17:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:17:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:17:56.190747 5195 mini_ranger.cc:160] Time spent starting Ranger: real 7.990s user 0.001s sys 0.008s
Sep 28, 2026 9:18:16 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:18:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:18:16 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:18:16 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:18:17 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:18:17.709803 5195 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangerkms_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/rangerkms_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.18.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.18.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:18:17 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630297, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:18:17.810832 5195 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:18:17.820217 5195 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:18:18.706558 5195 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:18:18.706662 5195 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms
I20260928 21:18:18.706746 5195 mini_ranger_kms.cc:227] Using postgres at 127.5.18.212:54555
2026-09-28 21:18:18,862 [I] DB FLAVOR :POSTGRES
2026-09-28 21:18:18,862 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:18:18,862 [I] Checking connection
2026-09-28 21:18:18,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:18:19,122 [I] connection success
2026-09-28 21:18:19,122 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:18:19,122 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:18:19,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:18:19,356 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:18:19,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:18:19,356 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:18:19,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:54555/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:18:19,615 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:18:20.076174 5195 mini_ranger_kms.cc:310] Created kms service
I20260928 21:18:20.270515 5195 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:18:20.317545 5195 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:18:20 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630297, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:20.723366 5195 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:18:20.723531 5195 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:18:20.723560 5195 mini_ranger_kms.cc:240] Using host: 127.5.18.212
I20260928 21:18:20.727475 5195 mini_ranger_kms.cc:276] Ranger KMS PID: 16331
I20260928 21:18:20.727589 5195 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.18.212:49595
Sep 28, 2026 9:18:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:18:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:18:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:18:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:18:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:18:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:18:22.745178 5195 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.934s user 0.017s sys 0.121s
I20260928 21:18:22.745299 5195 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:18:22.745404 5195 mini_ranger_kms.cc:401] 127.5.18.212:49595/kms/v1/keys
Sep 28 21:18:22 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:18:23 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
Sep 28 21:18:23 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630302, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
Sep 28 21:18:26 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:18:26.109077 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43567
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:43567
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:49595/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:18:26.222723 16382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:18:26.223060 16382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:18:26.223127 16382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:18:26.227270 16382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:18:26.227355 16382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:18:26.227387 16382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:18:26.227414 16382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:18:26.227440 16382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:18:26.233155 16382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.18.212:49595/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:43567
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:18:26.234344 16382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:18:26.235419 16382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:18:26.244508 16387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:26.244568 16382 server_base.cc:1034] running on GCE node
W20260928 21:18:26.244709 16388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:26.244508 16390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:26.245463 16382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:18:26.246655 16382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:18:26.247836 16382 hybrid_clock.cc:648] HybridClock initialized: now 1790630306247814 us; error 47 us; skew 500 ppm
Sep 28 21:18:26 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630306, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:18:26.255849 16382 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:18:26.256878 16382 webserver.cc:460] Webserver started at http://127.5.18.254:39261/ using document root <none> and password file <none>
I20260928 21:18:26.257412 16382 fs_manager.cc:359] Metadata directory not provided
I20260928 21:18:26.257469 16382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:18:26.257709 16382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:18:26 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630306, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:26.312068 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "c5246a8470d7494187899deb9399be93"
format_stamp: "Formatted at 2026-09-28 21:18:26 on dist-test-slave-rc9f"
server_key: "9391a177bb85c894b63a63b51a4e3375"
server_key_iv: "6672559f8176ded2ddb30912b6969c5a"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:26.312803 16382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "c5246a8470d7494187899deb9399be93"
format_stamp: "Formatted at 2026-09-28 21:18:26 on dist-test-slave-rc9f"
server_key: "9391a177bb85c894b63a63b51a4e3375"
server_key_iv: "6672559f8176ded2ddb30912b6969c5a"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:26.317777 16382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 21:18:26.331869 16397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:26.333269 16382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:18:26.333402 16382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "c5246a8470d7494187899deb9399be93"
format_stamp: "Formatted at 2026-09-28 21:18:26 on dist-test-slave-rc9f"
server_key: "9391a177bb85c894b63a63b51a4e3375"
server_key_iv: "6672559f8176ded2ddb30912b6969c5a"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:26.333541 16382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:18:26.353055 16382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:18:26.356076 16382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:18:26.356266 16382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:18:26.363662 16382 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:43567
I20260928 21:18:26.363719 16449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:43567 every 8 connection(s)
I20260928 21:18:26.364648 16382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:18:26.367298 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 16382
I20260928 21:18:26.367411 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:18:26.367683 5195 external_mini_cluster.cc:1428] Setting key b9bb8b5d91afe2be9c10499f3064195f
I20260928 21:18:26.368250 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:18:26.375792 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: Bootstrap starting.
Sep 28 21:18:26 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:18:26.378674 16450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: Neither blocks nor log segments found. Creating new log.
I20260928 21:18:26.380077 16450 log.cc:824] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: Log is configured to *not* fsync() on all Append() calls
I20260928 21:18:26.380918 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:26.369039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.5.18.254:43567)
0928 21:18:26.369438 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:26.369457 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:26.369659 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:18:26.370424 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:26.370440 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:26.370762 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:26.371061 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:18:26.371828 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:26.372804 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:26.373675 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:26.374133 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:26.374287 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:26.376636 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:18:26.376641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:18:26.376654 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:18:26.376680 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:18:26.378174 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:26.378575 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:26.378580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:26.378584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:26.378683 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:26.378936 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:26.378940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:26.378941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:26.379332 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:18:26.379501 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:26.379777 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 21:18:26.380059 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260928 21:18:26.382467 16450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: No bootstrap required, opened a new log
I20260928 21:18:26.386574 16450 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } }
I20260928 21:18:26.386910 16450 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:18:26.386984 16450 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5246a8470d7494187899deb9399be93, State: Initialized, Role: FOLLOWER
I20260928 21:18:26.387488 16450 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } }
I20260928 21:18:26.387655 16450 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:18:26.387732 16450 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:18:26.387833 16450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:18:26.389070 16450 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } }
I20260928 21:18:26.389448 16450 leader_election.cc:302] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5246a8470d7494187899deb9399be93; no voters:
I20260928 21:18:26.389858 16450 leader_election.cc:288] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:18:26.389959 16455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:18:26.390216 16455 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [term 1 LEADER]: Becoming Leader. State: Replica: c5246a8470d7494187899deb9399be93, State: Running, Role: LEADER
I20260928 21:18:26.390548 16455 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } }
I20260928 21:18:26.391240 16450 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:18:26.391793 16455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5246a8470d7494187899deb9399be93. Latest consensus state: current_term: 1 leader_uuid: "c5246a8470d7494187899deb9399be93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } } }
I20260928 21:18:26.392624 16455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [sys.catalog]: This master's current role is: LEADER
I20260928 21:18:26.392521 16457 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5246a8470d7494187899deb9399be93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5246a8470d7494187899deb9399be93" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43567 } } }
I20260928 21:18:26.392946 16457 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93 [sys.catalog]: This master's current role is: LEADER
I20260928 21:18:26.393656 16465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:18:26.393884 16450 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:18:26.400018 16465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:18:26.408378 16465 catalog_manager.cc:1347] Generated new cluster ID: 1ec105011c894b0db6cb2f10a786177b
I20260928 21:18:26.408516 16465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:18:26.417204 16465 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:18:26.418639 16465 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:18:26.427192 16465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: Generated new TSK 0
I20260928 21:18:26.428191 16465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:18:27.133353 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:49595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:18:27.268417 16495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:18:27.268824 16495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:18:27.268888 16495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:18:27.274972 16495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:18:27.275151 16495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:18:27.275291 16495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:18:27.281656 16495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.18.212:49595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:18:27.282743 16495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:18:27.283859 16495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:18:27.295985 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:27.296000 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:27.296499 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:27.296533 16495 server_base.cc:1034] running on GCE node
I20260928 21:18:27.297024 16495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:18:27.297727 16495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:18:27.298929 16495 hybrid_clock.cc:648] HybridClock initialized: now 1790630307298900 us; error 42 us; skew 500 ppm
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:18:27.302676 16495 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:18:27.303771 16495 webserver.cc:460] Webserver started at http://127.5.18.193:40773/ using document root <none> and password file <none>
I20260928 21:18:27.304454 16495 fs_manager.cc:359] Metadata directory not provided
I20260928 21:18:27.304541 16495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:18:27.304773 16495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:27.323392 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "570db1f649604cc1849fe775d9527cfc"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "d265fbecca7480aa8ccdc2992907ff68"
server_key_iv: "011b8f2bbac6d16a6f46f8d99391fcd0"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.324091 16495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "570db1f649604cc1849fe775d9527cfc"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "d265fbecca7480aa8ccdc2992907ff68"
server_key_iv: "011b8f2bbac6d16a6f46f8d99391fcd0"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.328203 16495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:18:27.358059 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.364282 16495 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260928 21:18:27.364450 16495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "570db1f649604cc1849fe775d9527cfc"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "d265fbecca7480aa8ccdc2992907ff68"
server_key_iv: "011b8f2bbac6d16a6f46f8d99391fcd0"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.364605 16495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:18:27.391376 16495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:18:27.395339 16495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:18:27.395593 16495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:18:27.396332 16495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:18:27.397457 16495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:18:27.397603 16495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.397740 16495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:18:27.397835 16495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.415252 16495 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:33189
I20260928 21:18:27.416615 16495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:18:27.417328 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:33189 every 8 connection(s)
I20260928 21:18:27.425473 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 16495
I20260928 21:18:27.425637 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:18:27.425992 5195 external_mini_cluster.cc:1428] Setting key f84fd1c6e05eaa80a6e7e8b3032dd542
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:18:27.438119 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:27.421888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43567 (local address 127.5.18.193:57939)
0928 21:18:27.422474 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:27.422532 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:27.423668 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:27.424330 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:27.424339 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:27.424882 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:18:27.425640 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:27.425663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.426889 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.426892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:27.428358 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:27.428369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.429193 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.429197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:27.429421 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:27.430181 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:18:27.430204 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:18:27.432109 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:18:27.434519 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:27.434526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:27.434548 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:27.434924 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:27.435255 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:27.435258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:27.435260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:27.435467 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:27.435962 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:18:27.435969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:18:27.436351 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 21:18:27.436561 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 21:18:27.436934 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":168,"threads_started":1}
I20260928 21:18:27.438897 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:27.421294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57939 (local address 127.5.18.254:43567)
0928 21:18:27.421601 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:27.421609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:27.422657 (+ 1048us) server_negotiation.cc:407] Connection header received
0928 21:18:27.423940 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:27.423944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:27.424010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:27.424124 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:18:27.425837 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.426723 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.428503 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.428870 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.428952 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:27.432353 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:18:27.432357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:18:27.432360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:18:27.432390 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:18:27.434362 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:27.435060 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:27.435064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:27.435066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:27.435159 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:27.435582 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:27.435585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:27.435587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:27.435824 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:18:27.435909 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:27.438583 (+ 2674us) server_negotiation.cc:299] Negotiation successful
0928 21:18:27.438728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:18:27.440039 16628 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43567
I20260928 21:18:27.440351 16628 heartbeater.cc:464] Registering TS with master...
I20260928 21:18:27.441155 16628 heartbeater.cc:511] Master 127.5.18.254:43567 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:18:27.443116 16414 ts_manager.cc:194] Registered new tserver with Master: 570db1f649604cc1849fe775d9527cfc (127.5.18.193:33189)
I20260928 21:18:27.444695 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:57939
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:18:27.465085 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:49595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:18:27.626808 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:18:27.627287 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:18:27.627414 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:18:27.633628 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:18:27.633715 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:18:27.633803 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:18:27.641963 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.18.212:49595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:18:27.643514 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:18:27.644935 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:18:27.658147 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:27.658157 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:27.658576 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:27.659071 16634 server_base.cc:1034] running on GCE node
I20260928 21:18:27.659590 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:18:27.660444 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:18:27.661656 16634 hybrid_clock.cc:648] HybridClock initialized: now 1790630307661623 us; error 51 us; skew 500 ppm
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:18:27.670296 16634 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:18:27.671789 16634 webserver.cc:460] Webserver started at http://127.5.18.194:43089/ using document root <none> and password file <none>
I20260928 21:18:27.672638 16634 fs_manager.cc:359] Metadata directory not provided
I20260928 21:18:27.672744 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:18:27.673089 16634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:27.697888 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "f748b48e25ab4649b0c7e40ec332fc1b"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "db338eed736fb24c36947c02e9f9d6bc"
server_key_iv: "93259bc03b1ab97f7fd9f30cc514a147"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.699651 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "f748b48e25ab4649b0c7e40ec332fc1b"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "db338eed736fb24c36947c02e9f9d6bc"
server_key_iv: "93259bc03b1ab97f7fd9f30cc514a147"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.705386 16634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 21:18:27.723338 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.724844 16634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:18:27.724972 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "f748b48e25ab4649b0c7e40ec332fc1b"
format_stamp: "Formatted at 2026-09-28 21:18:27 on dist-test-slave-rc9f"
server_key: "db338eed736fb24c36947c02e9f9d6bc"
server_key_iv: "93259bc03b1ab97f7fd9f30cc514a147"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:27.725102 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:18:27.739243 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:18:27.742184 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:18:27.742352 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:18:27.742926 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:18:27.743778 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:18:27.743826 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.743876 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:18:27.743902 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:27.770761 16634 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:36991
I20260928 21:18:27.772182 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:18:27.774132 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 16634
I20260928 21:18:27.774252 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:18:27.774413 16763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:36991 every 8 connection(s)
I20260928 21:18:27.774577 5195 external_mini_cluster.cc:1428] Setting key f119a4c7594598661cbe5628c3d3fc96
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:18:27.798578 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:27.777455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:43703 (local address 127.5.18.254:43567)
0928 21:18:27.777608 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:27.777612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:27.777628 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:18:27.778251 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:27.778254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:27.778309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:27.778360 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:18:27.780124 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.781169 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.783778 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.784274 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.784391 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:27.793881 (+ 9490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:18:27.793886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:18:27.793889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:18:27.793926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:18:27.795841 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:27.796580 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:27.796584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:27.796586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:27.796681 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:27.797130 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:27.797133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:27.797135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:27.797378 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:18:27.797458 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:27.798242 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 21:18:27.798388 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:18:27.800346 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:27.776298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43567 (local address 127.5.18.194:43703)
0928 21:18:27.776887 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:27.776948 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:27.777989 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:27.778819 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:27.778829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:27.779187 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:18:27.779904 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:27.779925 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.781415 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.781419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:27.783546 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:27.783587 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:27.784736 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:27.784741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:27.785058 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:27.785800 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:18:27.785823 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:18:27.793603 (+ 7780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:18:27.796013 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:27.796020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:27.796042 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:27.796440 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:27.796799 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:27.796802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:27.796804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:27.797012 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:27.797539 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:18:27.797547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:18:27.797944 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 21:18:27.798826 (+ 882us) client_negotiation.cc:240] Negotiation successful
0928 21:18:27.799205 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":144,"threads_started":1}
I20260928 21:18:27.801852 16764 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43567
I20260928 21:18:27.802286 16764 heartbeater.cc:464] Registering TS with master...
I20260928 21:18:27.803052 16764 heartbeater.cc:511] Master 127.5.18.254:43567 requested a full tablet report, sending...
I20260928 21:18:27.813998 16414 ts_manager.cc:194] Registered new tserver with Master: f748b48e25ab4649b0c7e40ec332fc1b (127.5.18.194:36991)
I20260928 21:18:27.814881 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:43703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:18:27.822427 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:49595/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:18:27 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
W20260928 21:18:27.987239 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:18:27.987707 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:18:27.987812 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:18:27.993963 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:18:27.994122 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:18:27.994254 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:18:28.002156 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.18.212:49595/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:18:28.003925 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:18:28.005388 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:18:28.019315 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:28.019681 16769 server_base.cc:1034] running on GCE node
W20260928 21:18:28.020241 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:18:28.020553 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:18:28.021134 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:18:28.021996 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:18:28.023286 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790630308023204 us; error 98 us; skew 500 ppm
Sep 28 21:18:28 dist-test-slave-rc9f krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:18:28.027062 16769 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:18:28.028183 16769 webserver.cc:460] Webserver started at http://127.5.18.195:39949/ using document root <none> and password file <none>
I20260928 21:18:28.028885 16769 fs_manager.cc:359] Metadata directory not provided
I20260928 21:18:28.028956 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:18:28.029181 16769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:18:28 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:28.048560 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "196c5d8f10494d5293e5efa8a5d644df"
format_stamp: "Formatted at 2026-09-28 21:18:28 on dist-test-slave-rc9f"
server_key: "b2261d94d349e02f0f6e77e21315b3c7"
server_key_iv: "7ecb541b896a321fcf294a94f5f667db"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:28.049271 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "196c5d8f10494d5293e5efa8a5d644df"
format_stamp: "Formatted at 2026-09-28 21:18:28 on dist-test-slave-rc9f"
server_key: "b2261d94d349e02f0f6e77e21315b3c7"
server_key_iv: "7ecb541b896a321fcf294a94f5f667db"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:28.056532 16769 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 21:18:28.074738 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:28.076586 16769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 21:18:28.076740 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "196c5d8f10494d5293e5efa8a5d644df"
format_stamp: "Formatted at 2026-09-28 21:18:28 on dist-test-slave-rc9f"
server_key: "b2261d94d349e02f0f6e77e21315b3c7"
server_key_iv: "7ecb541b896a321fcf294a94f5f667db"
server_key_version: "kuduclusterkey@0"
I20260928 21:18:28.076874 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:18:28.103355 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:18:28.107213 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:18:28.107409 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:18:28.108104 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:18:28.109218 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:18:28.109308 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:28.109375 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:18:28.109405 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:18:28.119599 16769 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:33943
I20260928 21:18:28.120927 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:18:28.122526 16897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:33943 every 8 connection(s)
I20260928 21:18:28.127542 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 16769
I20260928 21:18:28.127672 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:18:28.127976 5195 external_mini_cluster.cc:1428] Setting key 980c37bef963ca0525445dc8393f99ed
Sep 28 21:18:28 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630308, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:18:28.140292 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.124908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:37399 (local address 127.5.18.254:43567)
0928 21:18:28.125030 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.125034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.126523 (+ 1489us) server_negotiation.cc:407] Connection header received
0928 21:18:28.127815 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.127820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.127883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.127935 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:18:28.129622 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.130351 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.132017 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.132352 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.132441 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.135857 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:18:28.135862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:18:28.135865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:18:28.135895 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:18:28.137698 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:28.138365 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:28.138369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:28.138371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:28.138462 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:28.138886 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:28.138889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:28.138891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:28.139142 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:18:28.139214 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.139955 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.140115 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:18:28.142360 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.125117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43567 (local address 127.5.18.195:37399)
0928 21:18:28.126338 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.126406 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.127534 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.128358 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.128368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.128725 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:18:28.129429 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.129451 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.130498 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.130501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.131882 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.131893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.132665 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.132669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.132881 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.133549 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:18:28.133619 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:18:28.135638 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:18:28.137832 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:28.137839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:28.137859 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:28.138250 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:28.138571 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:18:28.138574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:18:28.138576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:18:28.138775 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:18:28.139298 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:18:28.139306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:18:28.139661 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.140770 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.141162 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":122,"threads_started":1}
I20260928 21:18:28.143875 16898 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43567
I20260928 21:18:28.144152 16898 heartbeater.cc:464] Registering TS with master...
I20260928 21:18:28.144732 16898 heartbeater.cc:511] Master 127.5.18.254:43567 requested a full tablet report, sending...
I20260928 21:18:28.145941 16414 ts_manager.cc:194] Registered new tserver with Master: 196c5d8f10494d5293e5efa8a5d644df (127.5.18.195:33943)
I20260928 21:18:28.146692 16414 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:37399
I20260928 21:18:28.153668 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:18:28.165784 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.155946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.5.18.254:43567)
0928 21:18:28.156058 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.156062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.156641 (+ 579us) server_negotiation.cc:407] Connection header received
0928 21:18:28.156801 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.156804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.156849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.156894 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:18:28.157858 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.158677 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.159859 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.160188 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.160255 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.162044 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:18:28.162048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:18:28.162051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:18:28.162077 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:18:28.163911 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:28.164403 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:28.164406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:28.164408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:28.164488 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:18:28.164771 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:18:28.164774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:18:28.164776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:18:28.164986 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:18:28.165085 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.165438 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.165608 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:18:28.170079 16414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:18:28.174738 16414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:18:28.203984 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.190044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:36991 (local address 127.0.0.1:48596)
0928 21:18:28.192936 (+ 2892us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.192947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.193009 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.195884 (+ 2875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.195887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.195905 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:18:28.195994 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.196000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.199535 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.199538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.200971 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.200983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.203386 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.203391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.203514 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.203558 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.203684 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.203817 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2837,"thread_start_us":85,"threads_started":1}
I20260928 21:18:28.205844 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.190480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.5.18.194:36991)
0928 21:18:28.195256 (+ 4776us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.195271 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.195294 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:18:28.195367 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.195372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.195575 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.195741 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:18:28.198218 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.199400 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.202150 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.203046 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.203151 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.203316 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.205547 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.205694 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":129,"threads_started":1}
I20260928 21:18:28.206538 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.191236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:33189 (local address 127.0.0.1:37522)
0928 21:18:28.200976 (+ 9740us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.200991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.201107 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.201884 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.201888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.201908 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:18:28.202014 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.202022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.203382 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.203386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.204563 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.204575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.206121 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.206124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.206223 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.206248 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.206328 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.206395 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9676,"thread_start_us":78,"threads_started":1}
I20260928 21:18:28.206640 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.191380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.5.18.193:33189)
0928 21:18:28.200825 (+ 9445us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.200832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.201210 (+ 378us) server_negotiation.cc:407] Connection header received
0928 21:18:28.201296 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.201302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.201530 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.201722 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:18:28.202162 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.203246 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.204714 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.205799 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.205915 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.206058 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.206403 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.206504 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9321,"thread_start_us":81,"threads_started":1}
I20260928 21:18:28.208721 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.187487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:33943 (local address 127.0.0.1:53040)
0928 21:18:28.192547 (+ 5060us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.192607 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.192730 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.194942 (+ 2212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.194949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.194975 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:18:28.195114 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.195121 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.204613 (+ 9492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.204616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.205892 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.205901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.208335 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.208338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.208436 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.208457 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.208522 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.208598 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4873,"thread_start_us":105,"threads_started":1}
I20260928 21:18:28.209123 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.188472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.5.18.195:33943)
0928 21:18:28.193274 (+ 4802us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.193286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.193309 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:18:28.193383 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.193389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.193651 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.193796 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:18:28.196130 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.204470 (+ 8340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.207194 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.208023 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.208134 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.208273 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.208889 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.208989 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4676,"thread_start_us":163,"threads_started":1}
I20260928 21:18:28.209383 16698 tablet_service.cc:1467] Processing CreateTablet for tablet ae56346ce6874249bb2206e068af0b16 (DEFAULT_TABLE table=test-table [id=aec0e7d32ef345919128846085a1cfd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:18:28.209786 16556 tablet_service.cc:1467] Processing CreateTablet for tablet ae56346ce6874249bb2206e068af0b16 (DEFAULT_TABLE table=test-table [id=aec0e7d32ef345919128846085a1cfd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:18:28.210628 16698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56346ce6874249bb2206e068af0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:18:28.210755 16556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56346ce6874249bb2206e068af0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:18:28.216406 16832 tablet_service.cc:1467] Processing CreateTablet for tablet ae56346ce6874249bb2206e068af0b16 (DEFAULT_TABLE table=test-table [id=aec0e7d32ef345919128846085a1cfd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:18:28.217464 16832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae56346ce6874249bb2206e068af0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:18:28.228977 16918 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Bootstrap starting.
I20260928 21:18:28.231773 16918 tablet_bootstrap.cc:654] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Neither blocks nor log segments found. Creating new log.
I20260928 21:18:28.233276 16918 log.cc:824] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:18:28.234018 16919 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Bootstrap starting.
I20260928 21:18:28.234568 16917 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: Bootstrap starting.
I20260928 21:18:28.235576 16918 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: No bootstrap required, opened a new log
I20260928 21:18:28.235786 16918 ts_tablet_manager.cc:1397] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 21:18:28.240044 16919 tablet_bootstrap.cc:654] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Neither blocks nor log segments found. Creating new log.
I20260928 21:18:28.240018 16918 raft_consensus.cc:348] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.240499 16918 raft_consensus.cc:374] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:18:28.240593 16918 raft_consensus.cc:729] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f748b48e25ab4649b0c7e40ec332fc1b, State: Initialized, Role: FOLLOWER
I20260928 21:18:28.240944 16919 log.cc:824] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Log is configured to *not* fsync() on all Append() calls
I20260928 21:18:28.241258 16918 consensus_queue.cc:260] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.242410 16918 ts_tablet_manager.cc:1428] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 21:18:28.242807 16764 heartbeater.cc:503] Master 127.5.18.254:43567 was elected leader, sending a full tablet report...
I20260928 21:18:28.243046 16917 tablet_bootstrap.cc:654] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: Neither blocks nor log segments found. Creating new log.
I20260928 21:18:28.243952 16917 log.cc:824] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:18:28.247874 16919 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: No bootstrap required, opened a new log
I20260928 21:18:28.248114 16919 ts_tablet_manager.cc:1397] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260928 21:18:28.249678 16917 tablet_bootstrap.cc:492] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: No bootstrap required, opened a new log
I20260928 21:18:28.249931 16917 ts_tablet_manager.cc:1397] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260928 21:18:28.252440 16919 raft_consensus.cc:348] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.252774 16919 raft_consensus.cc:374] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:18:28.252842 16919 raft_consensus.cc:729] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196c5d8f10494d5293e5efa8a5d644df, State: Initialized, Role: FOLLOWER
I20260928 21:18:28.253465 16919 consensus_queue.cc:260] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.255573 16919 ts_tablet_manager.cc:1428] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 21:18:28.254958 16917 raft_consensus.cc:348] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.255991 16898 heartbeater.cc:503] Master 127.5.18.254:43567 was elected leader, sending a full tablet report...
I20260928 21:18:28.256119 16917 raft_consensus.cc:374] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:18:28.256230 16917 raft_consensus.cc:729] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570db1f649604cc1849fe775d9527cfc, State: Initialized, Role: FOLLOWER
I20260928 21:18:28.256785 16917 consensus_queue.cc:260] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.259473 16917 ts_tablet_manager.cc:1428] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260928 21:18:28.259883 16628 heartbeater.cc:503] Master 127.5.18.254:43567 was elected leader, sending a full tablet report...
I20260928 21:18:28.264187 16924 raft_consensus.cc:482] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:18:28.264357 16924 raft_consensus.cc:504] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.269901 16924 leader_election.cc:288] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f748b48e25ab4649b0c7e40ec332fc1b (127.5.18.194:36991), 570db1f649604cc1849fe775d9527cfc (127.5.18.193:33189)
I20260928 21:18:28.270283 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.265529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:33189 (local address 127.5.18.195:56719)
0928 21:18:28.265742 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.265759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.265837 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.266816 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.266819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.266835 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:18:28.266929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.266935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.267806 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.267809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.268897 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.268904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.269866 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.269869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.269983 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.270009 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.270077 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.270150 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":12}
I20260928 21:18:28.270342 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.265984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56719 (local address 127.5.18.193:33189)
0928 21:18:28.266577 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.266581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.266592 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:18:28.266628 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.266630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.266668 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.266724 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:18:28.267043 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.267698 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.269003 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.269634 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.269728 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.269802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.270176 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.270247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517}
I20260928 21:18:28.271119 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.265802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:42155 (local address 127.5.18.194:36991)
0928 21:18:28.265935 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.265940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.265959 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:18:28.266083 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.266087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.266135 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.266188 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:18:28.266581 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.267292 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.268890 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.269459 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.269559 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.269636 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.270940 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.271013 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:18:28.271636 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.265538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:36991 (local address 127.5.18.195:42155)
0928 21:18:28.265877 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.265889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:18:28.265956 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:18:28.266318 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:18:28.266320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:18:28.266337 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:18:28.266449 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.266454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.267431 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.267434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.268759 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:18:28.268769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.269735 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.269738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:18:28.270769 (+ 1031us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.270796 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:18:28.271405 (+ 609us) client_negotiation.cc:240] Negotiation successful
0928 21:18:28.271477 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":15,"thread_start_us":91,"threads_started":1}
I20260928 21:18:28.271701 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56346ce6874249bb2206e068af0b16" candidate_uuid: "196c5d8f10494d5293e5efa8a5d644df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "570db1f649604cc1849fe775d9527cfc" is_pre_election: true
I20260928 21:18:28.272006 16572 raft_consensus.cc:2454] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196c5d8f10494d5293e5efa8a5d644df in term 0.
I20260928 21:18:28.272325 16700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56346ce6874249bb2206e068af0b16" candidate_uuid: "196c5d8f10494d5293e5efa8a5d644df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" is_pre_election: true
I20260928 21:18:28.272617 16700 raft_consensus.cc:2454] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 196c5d8f10494d5293e5efa8a5d644df in term 0.
I20260928 21:18:28.272816 16787 leader_election.cc:302] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196c5d8f10494d5293e5efa8a5d644df, 570db1f649604cc1849fe775d9527cfc; no voters:
W20260928 21:18:28.273456 16765 tablet.cc:2369] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:18:28.273994 16924 raft_consensus.cc:2781] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:18:28.274075 16924 raft_consensus.cc:482] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:18:28.274118 16924 raft_consensus.cc:3037] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:18:28.275108 16924 raft_consensus.cc:504] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.275713 16700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56346ce6874249bb2206e068af0b16" candidate_uuid: "196c5d8f10494d5293e5efa8a5d644df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f748b48e25ab4649b0c7e40ec332fc1b"
I20260928 21:18:28.275713 16572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae56346ce6874249bb2206e068af0b16" candidate_uuid: "196c5d8f10494d5293e5efa8a5d644df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "570db1f649604cc1849fe775d9527cfc"
I20260928 21:18:28.275822 16700 raft_consensus.cc:3037] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:18:28.275844 16572 raft_consensus.cc:3037] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:18:28.276970 16700 raft_consensus.cc:2454] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196c5d8f10494d5293e5efa8a5d644df in term 1.
I20260928 21:18:28.276970 16572 raft_consensus.cc:2454] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 196c5d8f10494d5293e5efa8a5d644df in term 1.
I20260928 21:18:28.277311 16786 leader_election.cc:302] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 196c5d8f10494d5293e5efa8a5d644df, 570db1f649604cc1849fe775d9527cfc, f748b48e25ab4649b0c7e40ec332fc1b; no voters:
I20260928 21:18:28.277374 16924 leader_election.cc:288] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [CANDIDATE]: Term 1 election: Requested vote from peers f748b48e25ab4649b0c7e40ec332fc1b (127.5.18.194:36991), 570db1f649604cc1849fe775d9527cfc (127.5.18.193:33189)
I20260928 21:18:28.277459 16924 raft_consensus.cc:2781] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:18:28.277541 16924 raft_consensus.cc:686] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 1 LEADER]: Becoming Leader. State: Replica: 196c5d8f10494d5293e5efa8a5d644df, State: Running, Role: LEADER
I20260928 21:18:28.278069 16924 consensus_queue.cc:237] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } }
I20260928 21:18:28.281917 16414 catalog_manager.cc:5495] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df reported cstate change: term changed from 0 to 1, leader changed from <none> to 196c5d8f10494d5293e5efa8a5d644df (127.5.18.195). New cstate: current_term: 1 leader_uuid: "196c5d8f10494d5293e5efa8a5d644df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "196c5d8f10494d5293e5efa8a5d644df" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 33943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:18:28 dist-test-slave-rc9f krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630307, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:18:28.301628 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.297229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.5.18.195:33943)
0928 21:18:28.297356 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:18:28.297361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:18:28.297376 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:18:28.297462 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:18:28.297465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:18:28.297535 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:18:28.297685 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:18:28.298127 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.298833 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.300049 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:18:28.300322 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:18:28.300423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:18:28.300714 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:18:28.301205 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:18:28.301319 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:18:28.301372 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:18:28.301430 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:18:28.313555 16572 raft_consensus.cc:1261] T ae56346ce6874249bb2206e068af0b16 P 570db1f649604cc1849fe775d9527cfc [term 1 FOLLOWER]: Refusing update from remote peer 196c5d8f10494d5293e5efa8a5d644df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:18:28.313771 16700 raft_consensus.cc:1261] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 1 FOLLOWER]: Refusing update from remote peer 196c5d8f10494d5293e5efa8a5d644df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:18:28.314401 16928 consensus_queue.cc:1035] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [LEADER]: Connected to new peer: Peer: permanent_uuid: "f748b48e25ab4649b0c7e40ec332fc1b" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:18:28.314577 16924 consensus_queue.cc:1035] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [LEADER]: Connected to new peer: Peer: permanent_uuid: "570db1f649604cc1849fe775d9527cfc" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:18:28.325456 16932 mvcc.cc:205] Tried to move back new op lower bound from 7334421742843985920 to 7334421742710136832. Current Snapshot: MvccSnapshot[applied={T|T < 7334421742843985920}]
I20260928 21:18:28.337037 16414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:18:28.337251 16414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:18:28.339896 16414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5246a8470d7494187899deb9399be93: Sending DeleteTablet for 3 replicas of tablet ae56346ce6874249bb2206e068af0b16
I20260928 21:18:28.340842 16556 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56346ce6874249bb2206e068af0b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:18:28 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:37522
I20260928 21:18:28.340927 16832 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56346ce6874249bb2206e068af0b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:18:28 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:53040
I20260928 21:18:28.341224 16698 tablet_service.cc:1514] Processing DeleteTablet for tablet ae56346ce6874249bb2206e068af0b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:18:28 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:48596
I20260928 21:18:28.341747 16943 tablet_replica.cc:330] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: stopping tablet replica
I20260928 21:18:28.342471 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 16495
I20260928 21:18:28.346877 16941 tablet_replica.cc:330] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: stopping tablet replica
I20260928 21:18:28.347247 16941 raft_consensus.cc:2229] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:18:28.347631 16943 raft_consensus.cc:2229] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:18:28.347705 16941 raft_consensus.cc:2258] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:18:28.348125 16943 raft_consensus.cc:2258] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:18:28.351969 16943 ts_tablet_manager.cc:1905] T ae56346ce6874249bb2206e068af0b16 P f748b48e25ab4649b0c7e40ec332fc1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:18:28.354756 16941 ts_tablet_manager.cc:1905] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:18:28.355582 16400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:33189 (error 108)
I20260928 21:18:28.356194 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 16634
I20260928 21:18:28.358886 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.358515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:33189 (local address 127.0.0.1:37524)
0928 21:18:28.358722 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.358794 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260928 21:18:28.359180 16400 catalog_manager.cc:4584] TS 570db1f649604cc1849fe775d9527cfc (127.5.18.193:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae56346ce6874249bb2206e068af0b16: Network error: Client connection negotiation failed: client connection to 127.5.18.193:33189: connect: Connection refused (error 111)
I20260928 21:18:28.364075 16941 ts_tablet_manager.cc:1918] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:18:28.364274 16941 log.cc:1196] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/ae56346ce6874249bb2206e068af0b16
I20260928 21:18:28.364708 16941 ts_tablet_manager.cc:1939] T ae56346ce6874249bb2206e068af0b16 P 196c5d8f10494d5293e5efa8a5d644df: Deleting consensus metadata
I20260928 21:18:28.367865 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:18:28.367540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:36991 (local address 127.0.0.1:48604)
0928 21:18:28.367722 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:18:28.367775 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:36991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 21:18:28.368085 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 16769
I20260928 21:18:28.381343 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 16382
I20260928 21:18:28.389251 5195 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:18:28.733757 5195 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:18:28.733891 5195 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:18:29.068674 5195 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:18:29.068 UTC [15968] LOG: received smart shutdown request
2026-09-28 21:18:29.072 UTC [15968] LOG: background worker "logical replication launcher" (PID 15977) exited with exit code 1
2026-09-28 21:18:29.072 UTC [15972] LOG: shutting down
2026-09-28 21:18:29.236 UTC [15968] LOG: database system is shut down
2026-09-28T21:18:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45428 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:18:29 dist-test-slave-rc9f krb5kdc[16953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:18:29 dist-test-slave-rc9f krb5kdc[16953](info): set up 2 sockets
Sep 28 21:18:29 dist-test-slave-rc9f krb5kdc[16953](info): commencing operation
krb5kdc: starting...
W20260928 21:18:30.289409 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:18:30 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:18:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangeradmin_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangeradmin_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangerlookup_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangerlookup_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
I20260928 21:18:30.456892 5195 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/postgres -l logfile start
2026-09-28 21:18:32.864 UTC [16983] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:18:32.864 UTC [16983] LOG: listening on IPv4 address "127.5.18.212", port 58609
2026-09-28 21:18:32.868 UTC [16983] LOG: listening on Unix socket "/tmp/.s.PGSQL.58609"
2026-09-28 21:18:32.895 UTC [16986] LOG: database system was shut down at 2026-09-28 21:18:31 UTC
2026-09-28 21:18:32.898 UTC [16983] LOG: database system is ready to accept connections
I20260928 21:18:33.753257 5195 mini_postgres.cc:96] Postgres bound to 58609
2026-09-28 21:18:33.759 UTC [16994] FATAL: database "slave" does not exist
127.5.18.212:58609 - accepting connections
I20260928 21:18:33.759660 5195 mini_ranger.cc:161] Starting Ranger...
I20260928 21:18:33.773962 5195 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:18:34.228933 5195 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:18:34.229013 5195 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin
2026-09-28 21:18:34,495 [I] DB FLAVOR :POSTGRES
2026-09-28 21:18:34,495 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:18:34,495 [I] Checking connection..
2026-09-28 21:18:34,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:18:34,738 [I] Checking connection passed.
2026-09-28 21:18:34,738 [I] --------- Verifying version history table ---------
2026-09-28 21:18:34,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:18:34,992 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:18:34,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:18:35,228 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:18:35,229 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:18:35,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:18:35.465 UTC [17058] WARNING: there is no transaction in progress
2026-09-28 21:18:35,471 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:18:35,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:18:35,702 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:18:35,702 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:18:35,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:18:35,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:18:36,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rc9f.c.gcp-upstream.internal','N') ;"
2026-09-28 21:18:36,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:18:36,667 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:18:36,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:18:36,909 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:18:36,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:18:37,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:18:37,420 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:18:37,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:18:38.437 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.455 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.472 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.490 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.506 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.526 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.583 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.590 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.605 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.615 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.625 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.640 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.650 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.657 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.664 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:38.671 UTC [17166] WARNING: there is no transaction in progress
2026-09-28 21:18:39,321 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:18:39,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:18:39,583 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:18:39,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:18:39,838 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:18:39,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:18:40,079 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:18:40,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:18:40,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:18:40,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rc9f.c.gcp-upstream.internal';"
2026-09-28 21:18:40,815 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:18:40,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:18:41,050 [I] DB_PATCHES have already been applied
I20260928 21:18:41.055853 5195 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyuKJ5W/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:18:41.055926 5195 mini_ranger.cc:193] Using host: 127.5.18.212
I20260928 21:18:41.059180 5195 mini_ranger.cc:227] Ranger admin URL: http://127.5.18.212:55221
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:18:42.067706 5195 mini_ranger.cc:160] Time spent starting Ranger: real 8.308s user 0.000s sys 0.009s
Sep 28, 2026 9:19:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:19:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 9:19:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:19:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:19:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:19:03.848479 5195 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.18.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.18.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangerkms_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.18.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/rangerkms_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.18.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.18.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.18.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/HTTP_127.5.18.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:19:03 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:19:03.944578 5195 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:19:03.954394 5195 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:19:04.819123 5195 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:19:04.819207 5195 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms
I20260928 21:19:04.819223 5195 mini_ranger_kms.cc:227] Using postgres at 127.5.18.212:58609
2026-09-28 21:19:04,972 [I] DB FLAVOR :POSTGRES
2026-09-28 21:19:04,972 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:19:04,972 [I] Checking connection
2026-09-28 21:19:04,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:19:05,205 [I] connection success
2026-09-28 21:19:05,206 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:19:05,206 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:19:05,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:19:05,439 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:19:05,440 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:19:05,440 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:19:05,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.18.212:58609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:19:05,700 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:19:06.188512 5195 mini_ranger_kms.cc:310] Created kms service
I20260928 21:19:06.410084 5195 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:19:06.452764 5195 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:19:06 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630343, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:06.859236 5195 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:19:06.859405 5195 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyuKJ5W/build/debug/bin/postgresql.jar:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyuKJ5W/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:19:06.859431 5195 mini_ranger_kms.cc:240] Using host: 127.5.18.212
I20260928 21:19:06.863116 5195 mini_ranger_kms.cc:276] Ranger KMS PID: 17346
I20260928 21:19:06.863224 5195 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.18.212:52773
Sep 28, 2026 9:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:19:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:19:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:19:08.976667 5195 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.032s user 0.000s sys 0.138s
I20260928 21:19:08.976778 5195 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:19:08.976867 5195 mini_ranger_kms.cc:401] 127.5.18.212:52773/kms/v1/keys
Sep 28 21:19:09 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:19:09 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
Sep 28 21:19:10 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630349, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.18.212@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
Sep 28 21:19:12 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:19:12.437039 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45321
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:45321
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:12.543506 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:12.543780 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:12.543818 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:12.547859 17397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:12.547930 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:12.547947 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:12.547963 17397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:12.547977 17397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:12.553517 17397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.18.212:52773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:45321
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:12.554641 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:12.555684 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:12.564191 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:12.564301 17397 server_base.cc:1034] running on GCE node
W20260928 21:19:12.564211 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:12.564195 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:12.565063 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:12.566250 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:12.567404 17397 hybrid_clock.cc:648] HybridClock initialized: now 1790630352567386 us; error 34 us; skew 500 ppm
Sep 28 21:19:12 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:12.570233 17397 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:12.571163 17397 webserver.cc:460] Webserver started at http://127.5.18.254:37125/ using document root <none> and password file <none>
I20260928 21:19:12.571687 17397 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:12.571743 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:12.571945 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:19:12 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630352, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:12.634850 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "04183ce1998242dfbb2f1f4c174885a8"
format_stamp: "Formatted at 2026-09-28 21:19:12 on dist-test-slave-rc9f"
server_key: "e0698a574865ecc788420cdfef970406"
server_key_iv: "18d808d942143222785083ce5a66a176"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:12.635722 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "04183ce1998242dfbb2f1f4c174885a8"
format_stamp: "Formatted at 2026-09-28 21:19:12 on dist-test-slave-rc9f"
server_key: "e0698a574865ecc788420cdfef970406"
server_key_iv: "18d808d942143222785083ce5a66a176"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:12.640812 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 21:19:12.659005 17412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:12.660830 17397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:19:12.660989 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "04183ce1998242dfbb2f1f4c174885a8"
format_stamp: "Formatted at 2026-09-28 21:19:12 on dist-test-slave-rc9f"
server_key: "e0698a574865ecc788420cdfef970406"
server_key_iv: "18d808d942143222785083ce5a66a176"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:12.661135 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:12.675318 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:12.678382 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:12.678576 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:12.685880 17397 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:45321
I20260928 21:19:12.685897 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:45321 every 8 connection(s)
I20260928 21:19:12.686909 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:12.689734 17465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:12.694648 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17397
I20260928 21:19:12.694768 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:19:12.695017 5195 external_mini_cluster.cc:1428] Setting key ca43a07d624fc6eda26826f5c5bd2e2c
I20260928 21:19:12.695205 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Bootstrap starting.
I20260928 21:19:12.697853 17465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:12.698710 17465 log.cc:824] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:12.700472 17465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: No bootstrap required, opened a new log
Sep 28 21:19:12 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:12.704097 17465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:12.704329 17465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:12.704381 17465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04183ce1998242dfbb2f1f4c174885a8, State: Initialized, Role: FOLLOWER
I20260928 21:19:12.704783 17465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:12.704898 17465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:12.704959 17465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:12.705052 17465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:12.705940 17465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:12.706300 17465 leader_election.cc:302] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04183ce1998242dfbb2f1f4c174885a8; no voters:
I20260928 21:19:12.706643 17465 leader_election.cc:288] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:19:12.706750 17470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:12.706969 17470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 LEADER]: Becoming Leader. State: Replica: 04183ce1998242dfbb2f1f4c174885a8, State: Running, Role: LEADER
I20260928 21:19:12.707356 17470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:12.707845 17465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:12.708942 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04183ce1998242dfbb2f1f4c174885a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } } }
I20260928 21:19:12.709049 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:12.709260 17471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04183ce1998242dfbb2f1f4c174885a8. Latest consensus state: current_term: 1 leader_uuid: "04183ce1998242dfbb2f1f4c174885a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } } }
I20260928 21:19:12.709323 17471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:12.709378 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:12.696426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.5.18.254:45321)
0928 21:19:12.696856 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:12.696875 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:12.696917 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:19:12.697751 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:12.697775 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:12.698211 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:12.698565 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:12.699264 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:12.700231 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:12.701153 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:12.701688 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:12.701833 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:12.704368 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:12.704375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:12.704394 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:12.704425 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:12.706037 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:12.706535 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:12.706540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:12.706545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:12.706661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:12.707000 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:12.707003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:12.707005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:12.707517 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:12.707671 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:12.708112 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 21:19:12.708387 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260928 21:19:12.709820 17480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:12.709820 17465 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:19:12.715138 17480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:12.721380 17480 catalog_manager.cc:1347] Generated new cluster ID: 0971cbd58a624674a2723c5ee77c49dd
I20260928 21:19:12.721529 17480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:12.748421 17480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:19:12.749845 17480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:12.762326 17480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Generated new TSK 0
I20260928 21:19:12.763062 17480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:19:13.444952 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:13.585858 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:13.586159 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:13.586221 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:13.590324 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:13.590384 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:13.590510 17510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:13.596095 17510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:13.597201 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:13.598289 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:13.610082 17520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:13.610175 17518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:13.610239 17517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:13.610970 17510 server_base.cc:1034] running on GCE node
I20260928 21:19:13.611405 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:13.612087 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:13.613257 17510 hybrid_clock.cc:648] HybridClock initialized: now 1790630353613230 us; error 44 us; skew 500 ppm
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:13.616252 17510 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:13.617441 17510 webserver.cc:460] Webserver started at http://127.5.18.193:33235/ using document root <none> and password file <none>
I20260928 21:19:13.618212 17510 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:13.618310 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:13.618577 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:13.636245 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "1d58855e34b3406fb5b28136d90eeb87"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "39cd6317e89c3d812319b3f03739943b"
server_key_iv: "a1f3f141a5de3b6d00498279afe7ec28"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.636858 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "1d58855e34b3406fb5b28136d90eeb87"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "39cd6317e89c3d812319b3f03739943b"
server_key_iv: "a1f3f141a5de3b6d00498279afe7ec28"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.643805 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260928 21:19:13.661067 17527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:13.662978 17510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:13.663136 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "1d58855e34b3406fb5b28136d90eeb87"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "39cd6317e89c3d812319b3f03739943b"
server_key_iv: "a1f3f141a5de3b6d00498279afe7ec28"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.663293 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:13.680181 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:13.684257 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:13.684530 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:13.685295 17510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:13.686424 17510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:13.686527 17510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:13.686636 17510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:13.686703 17510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:13.696214 17510 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45863
I20260928 21:19:13.700381 17640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45863 every 8 connection(s)
I20260928 21:19:13.700802 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:13.703195 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17510
I20260928 21:19:13.703312 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:19:13.703619 5195 external_mini_cluster.cc:1428] Setting key 13e7493dc2b617ab093399da1d13be11
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:13.724009 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:13.703509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.193:33329)
0928 21:19:13.709528 (+ 6019us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:13.709629 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:13.710737 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:13.711857 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:13.711869 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:13.712256 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:13.712994 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:13.713016 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:13.714233 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:13.714236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:13.715372 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:13.715384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:13.716096 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:13.716099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:13.716283 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:13.716819 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:13.716835 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:13.718352 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:13.720486 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:13.720493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:13.720513 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:13.720885 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:13.721203 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:13.721206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:13.721208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:13.721402 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:13.721913 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:13.721920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:13.722265 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 21:19:13.722499 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 21:19:13.722878 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5714,"thread_start_us":150,"threads_started":1}
I20260928 21:19:13.725144 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:13.702998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:33329 (local address 127.5.18.254:45321)
0928 21:19:13.711263 (+ 8265us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:13.711271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:13.711290 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:19:13.711339 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:13.711342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:13.711394 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:13.711494 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:13.713175 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:13.714083 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:13.715508 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:13.715921 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:13.716041 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:13.718587 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:13.718590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:13.718593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:13.718621 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:13.720357 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:13.721010 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:13.721014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:13.721016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:13.721109 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:13.721508 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:13.721511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:13.721513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:13.721771 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:13.721860 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:13.724798 (+ 2938us) server_negotiation.cc:299] Negotiation successful
0928 21:19:13.724956 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8160,"thread_start_us":89,"threads_started":1}
I20260928 21:19:13.726377 17641 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:13.726656 17641 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:13.727255 17641 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:13.729118 17429 ts_manager.cc:194] Registered new tserver with Master: 1d58855e34b3406fb5b28136d90eeb87 (127.5.18.193:45863)
I20260928 21:19:13.730721 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:33329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:19:13.744108 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:19:13.865329 17649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:13.865746 17649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:13.865836 17649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:13.870994 17649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:13.871069 17649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:13.871151 17649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:13.877158 17649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:13.878327 17649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:13.879419 17649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:13.888867 17655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:13.888862 17658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:13.888998 17656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:13.889462 17649 server_base.cc:1034] running on GCE node
I20260928 21:19:13.890378 17649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:13.891229 17649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:13.892462 17649 hybrid_clock.cc:648] HybridClock initialized: now 1790630353892435 us; error 47 us; skew 500 ppm
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:13.895303 17649 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:13.896203 17649 webserver.cc:460] Webserver started at http://127.5.18.194:39469/ using document root <none> and password file <none>
I20260928 21:19:13.896755 17649 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:13.896821 17649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:13.897053 17649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:19:13 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:13.914784 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "6c90c384beda4173a6ef8e8042163589"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "5da3c40115860fb7025eb17e6b3f944a"
server_key_iv: "dbaada302282de1055d8e5317af7754b"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.915423 17649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "6c90c384beda4173a6ef8e8042163589"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "5da3c40115860fb7025eb17e6b3f944a"
server_key_iv: "dbaada302282de1055d8e5317af7754b"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.919260 17649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:19:13.958058 17665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:13.963299 17649 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.000s sys 0.004s
I20260928 21:19:13.963567 17649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "6c90c384beda4173a6ef8e8042163589"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "5da3c40115860fb7025eb17e6b3f944a"
server_key_iv: "dbaada302282de1055d8e5317af7754b"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:13.963809 17649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:13.985001 17649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:13.989249 17649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:13.989516 17649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:13.990347 17649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:13.991613 17649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:13.991735 17649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:13.991837 17649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:13.991910 17649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.003621 17649 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:34947
I20260928 21:19:14.007867 17778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:34947 every 8 connection(s)
I20260928 21:19:14.008882 17649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:19:14.011332 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17649
I20260928 21:19:14.011452 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:19:14.011771 5195 external_mini_cluster.cc:1428] Setting key 7789ee2b3fac259d28749b544115be60
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:14.040225 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.016383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:44243 (local address 127.5.18.254:45321)
0928 21:19:14.016533 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:14.016538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:14.018551 (+ 2013us) server_negotiation.cc:407] Connection header received
0928 21:19:14.024709 (+ 6158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:14.024714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:14.024795 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:14.024876 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:14.028081 (+ 3205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.028952 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.030777 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.031168 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.031253 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:14.034445 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:14.034449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:14.034452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:14.034484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:14.036249 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:14.038287 (+ 2038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:14.038292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:14.038294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:14.038398 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:14.038858 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:14.038862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:14.038864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:14.039103 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:14.039182 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:14.039903 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 21:19:14.040048 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:19:14.042171 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.016982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.194:44243)
0928 21:19:14.018362 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:14.018433 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:14.019718 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:14.026627 (+ 6909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:14.026640 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:14.027065 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:14.027875 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.027899 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.029126 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.029129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:14.030625 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.030638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.031479 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.031482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:14.031702 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:14.032439 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:14.032463 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:14.034230 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:14.037002 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:14.037009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:14.037030 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:14.037410 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:14.038512 (+ 1102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:14.038515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:14.038517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:14.038736 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:14.039249 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:14.039256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:14.039610 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 21:19:14.040668 (+ 1058us) client_negotiation.cc:240] Negotiation successful
0928 21:19:14.041065 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"thread_start_us":179,"threads_started":1}
I20260928 21:19:14.043682 17779 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:14.043931 17779 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:14.044484 17779 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:14.045732 17429 ts_manager.cc:194] Registered new tserver with Master: 6c90c384beda4173a6ef8e8042163589 (127.5.18.194:34947)
I20260928 21:19:14.046537 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:44243
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:19:14.074743 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:14.231383 17784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:14.231793 17784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:14.231855 17784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:14.238137 17784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:14.238276 17784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:14.238386 17784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:14.246316 17784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:14.247900 17784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:14.249332 17784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630353, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
W20260928 21:19:14.268383 17792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:14.268816 17789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:14.268841 17790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:14.269261 17784 server_base.cc:1034] running on GCE node
I20260928 21:19:14.269846 17784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:14.270692 17784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:14.278107 17784 hybrid_clock.cc:648] HybridClock initialized: now 1790630354277889 us; error 233 us; skew 500 ppm
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:14.282321 17784 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:14.283603 17784 webserver.cc:460] Webserver started at http://127.5.18.195:42863/ using document root <none> and password file <none>
I20260928 21:19:14.284405 17784 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:14.284497 17784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:14.284734 17784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:14.308882 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "c7614629636045ebb97350ad45c9c6fc"
format_stamp: "Formatted at 2026-09-28 21:19:14 on dist-test-slave-rc9f"
server_key: "2a8fbc392361561dc0f7eef200ccc267"
server_key_iv: "3d9738dc73784e42ba8b971a19067134"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:14.309906 17784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "c7614629636045ebb97350ad45c9c6fc"
format_stamp: "Formatted at 2026-09-28 21:19:14 on dist-test-slave-rc9f"
server_key: "2a8fbc392361561dc0f7eef200ccc267"
server_key_iv: "3d9738dc73784e42ba8b971a19067134"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:14.319432 17784 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.002s
I20260928 21:19:14.362089 17799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.364085 17784 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.003s sys 0.000s
I20260928 21:19:14.364234 17784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "c7614629636045ebb97350ad45c9c6fc"
format_stamp: "Formatted at 2026-09-28 21:19:14 on dist-test-slave-rc9f"
server_key: "2a8fbc392361561dc0f7eef200ccc267"
server_key_iv: "3d9738dc73784e42ba8b971a19067134"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:14.364382 17784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:14.403013 17784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:14.406705 17784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:14.406944 17784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:14.407678 17784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:14.408706 17784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:14.408793 17784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.408871 17784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:14.408916 17784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.419615 17784 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:34905
I20260928 21:19:14.419664 17912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:34905 every 8 connection(s)
I20260928 21:19:14.420847 17784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:19:14.428755 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17784
I20260928 21:19:14.428892 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:19:14.429183 5195 external_mini_cluster.cc:1428] Setting key 00a59613094b7c37eaddc4d82ae6e84d
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:14.438026 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.423038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46003 (local address 127.5.18.254:45321)
0928 21:19:14.423189 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:14.423194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:14.424315 (+ 1121us) server_negotiation.cc:407] Connection header received
0928 21:19:14.425623 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:14.425627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:14.425686 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:14.425778 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:14.427376 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.428200 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.430298 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.430663 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.430747 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:14.433876 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:14.433880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:14.433882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:14.433912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:14.435547 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:14.436178 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:14.436182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:14.436184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:14.436293 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:14.436671 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:14.436674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:14.436676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:14.436919 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:14.437022 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:14.437714 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 21:19:14.437856 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:19:14.439101 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.423548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.195:46003)
0928 21:19:14.424131 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:14.424192 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:14.425322 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:14.425994 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:14.426005 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:14.426397 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:14.427175 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.427199 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.428385 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.428389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:14.429952 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.430145 (+ 193us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.430968 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.430972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:14.431196 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:14.431878 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:14.431901 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:14.433668 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:14.435676 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:14.435683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:14.435703 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:14.436069 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:14.436384 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:14.436387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:14.436389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:14.436574 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:14.437018 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:14.437026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:14.437402 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 21:19:14.437666 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 21:19:14.438018 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":160,"threads_started":1}
I20260928 21:19:14.440439 17913 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:14.440733 17913 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:14.441388 17913 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:14.442783 17429 ts_manager.cc:194] Registered new tserver with Master: c7614629636045ebb97350ad45c9c6fc (127.5.18.195:34905)
I20260928 21:19:14.443547 17429 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:46003
I20260928 21:19:14.444293 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:19:14.444977 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17510
I20260928 21:19:14.451402 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17649
I20260928 21:19:14.458168 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17784
I20260928 21:19:14.465389 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17397
I20260928 21:19:14.471302 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45321
--webserver_interface=127.5.18.254
--webserver_port=37125
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:45321
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:14.581182 17916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:14.581487 17916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:14.581527 17916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:14.585966 17916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:14.586051 17916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:14.586071 17916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:14.586086 17916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:14.586100 17916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:14.591898 17916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:45321
--ranger_config_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:45321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=37125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:14.593070 17916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:14.594250 17916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:14.602602 17924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:14.602627 17921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:14.602882 17916 server_base.cc:1034] running on GCE node
W20260928 21:19:14.602986 17922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:14.603502 17916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:14.604609 17916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:14.605759 17916 hybrid_clock.cc:648] HybridClock initialized: now 1790630354605743 us; error 29 us; skew 500 ppm
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:14.608557 17916 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:14.609458 17916 webserver.cc:460] Webserver started at http://127.5.18.254:37125/ using document root <none> and password file <none>
I20260928 21:19:14.610049 17916 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:14.610122 17916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:14.680253 17916 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:14.682632 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.683967 17916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:19:14.684091 17916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "04183ce1998242dfbb2f1f4c174885a8"
format_stamp: "Formatted at 2026-09-28 21:19:12 on dist-test-slave-rc9f"
server_key: "e0698a574865ecc788420cdfef970406"
server_key_iv: "18d808d942143222785083ce5a66a176"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:14.684713 17916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:14.714828 17916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:14.717655 17916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:14.717828 17916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:14.724892 17916 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:45321
I20260928 21:19:14.724912 17984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:45321 every 8 connection(s)
I20260928 21:19:14.726029 17916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:14.728755 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17916
I20260928 21:19:14.729163 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:45863
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:45321
--webserver_port=33235
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:14.730412 17985 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:19:14.732071 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Bootstrap starting.
I20260928 21:19:14.748474 17985 log.cc:824] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:14.755419 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:14.755805 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Bootstrap complete.
I20260928 21:19:14.759851 17985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:14.760278 17985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04183ce1998242dfbb2f1f4c174885a8, State: Initialized, Role: FOLLOWER
I20260928 21:19:14.760758 17985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:14.760881 17985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:14.760948 17985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:14.761040 17985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:19:14.763120 17985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:14.763480 17985 leader_election.cc:302] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04183ce1998242dfbb2f1f4c174885a8; no voters:
I20260928 21:19:14.763800 17985 leader_election.cc:288] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:19:14.763906 17989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:19:14.764140 17989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [term 2 LEADER]: Becoming Leader. State: Replica: 04183ce1998242dfbb2f1f4c174885a8, State: Running, Role: LEADER
I20260928 21:19:14.764498 17989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } }
I20260928 21:19:14.765086 17985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:14.765962 17991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04183ce1998242dfbb2f1f4c174885a8. Latest consensus state: current_term: 2 leader_uuid: "04183ce1998242dfbb2f1f4c174885a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } } }
I20260928 21:19:14.766085 17991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:14.766358 17990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "04183ce1998242dfbb2f1f4c174885a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04183ce1998242dfbb2f1f4c174885a8" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 45321 } } }
I20260928 21:19:14.766440 17990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:14.766430 17998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:14.769924 17998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:14.770479 17985 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:19:14.770628 17998 catalog_manager.cc:1259] Loaded cluster ID: 0971cbd58a624674a2723c5ee77c49dd
I20260928 21:19:14.770677 17998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:14.773191 17998 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:14.773967 17998 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Loaded TSK: 0
I20260928 21:19:14.778113 17998 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:19:14.868549 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:14.868834 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:14.868896 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:14.873265 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:14.873353 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:14.873447 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:14.881448 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:45863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=33235
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:14.883044 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:14.884279 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:14.894045 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:14.894235 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:14.894593 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:14.894877 17987 server_base.cc:1034] running on GCE node
I20260928 21:19:14.895473 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:14.896169 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:14.897338 17987 hybrid_clock.cc:648] HybridClock initialized: now 1790630354897307 us; error 45 us; skew 500 ppm
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:14.900158 17987 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:14.901064 17987 webserver.cc:460] Webserver started at http://127.5.18.193:33235/ using document root <none> and password file <none>
I20260928 21:19:14.901823 17987 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:14.901980 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:14.925827 17987 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:14.932196 18033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.934156 17987 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260928 21:19:14.934324 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "1d58855e34b3406fb5b28136d90eeb87"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "39cd6317e89c3d812319b3f03739943b"
server_key_iv: "a1f3f141a5de3b6d00498279afe7ec28"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:14.935115 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:14.965783 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:14.969846 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:14.970134 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:14.970937 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:14.972045 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:14.972142 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.972213 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:14.972246 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:14.983529 17987 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45863
I20260928 21:19:14.983568 18146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45863 every 8 connection(s)
I20260928 21:19:14.984726 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:14.989094 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 17987
I20260928 21:19:14.989609 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:34947
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:45321
--webserver_port=39469
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:19:14 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630354, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:15.015403 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.987245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.193:51993)
0928 21:19:14.988489 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:14.988547 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:14.992224 (+ 3677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:14.994449 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:14.994459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:14.994846 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:14.995494 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.995513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.997443 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.997447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:14.998894 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:14.998917 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.000186 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.000190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:15.000387 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.001113 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:15.001133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:15.002926 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:15.005592 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.005600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.005620 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.006007 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.006441 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.006444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.006446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.006626 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.013162 (+ 6536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:15.013172 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:15.013644 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 21:19:15.013918 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 21:19:15.014318 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":134,"threads_started":1}
I20260928 21:19:15.015529 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:14.987097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:51993 (local address 127.5.18.254:45321)
0928 21:19:14.987741 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:14.987766 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:14.991262 (+ 3496us) server_negotiation.cc:407] Connection header received
0928 21:19:14.992948 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:14.992963 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:14.993455 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:14.993878 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:14.995688 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.997276 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.999053 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:14.999774 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:14.999943 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.003196 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:15.003204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:15.003224 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:15.003264 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:15.005417 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.006158 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.006165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.006172 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.006328 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.006740 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.006744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.006746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.007301 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:15.013769 (+ 6468us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:15.014158 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 21:19:15.014536 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":107,"threads_started":1}
I20260928 21:19:15.018637 18147 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:15.018908 18147 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.019544 18147 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
W20260928 21:19:15.022167 18147 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:45321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:19:15.022315 18147 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.022537 18147 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:15.023658 18147 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.023864 18147 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
W20260928 21:19:15.028429 18147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:19:15.175639 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:15.176084 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:15.176160 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:15.182598 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:15.182703 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:15.182802 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:15.191411 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:34947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=39469
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:15.193004 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:15.194527 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:15.208158 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:15.208686 18151 server_base.cc:1034] running on GCE node
W20260928 21:19:15.208282 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:15.208169 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:15.209281 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:15.210006 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:15.211184 18151 hybrid_clock.cc:648] HybridClock initialized: now 1790630355211158 us; error 40 us; skew 500 ppm
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:15.213994 18151 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:15.215097 18151 webserver.cc:460] Webserver started at http://127.5.18.194:39469/ using document root <none> and password file <none>
I20260928 21:19:15.215932 18151 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:15.216110 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:15.247085 18151 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 21:19:15.250480 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.252199 18151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:15.252327 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "6c90c384beda4173a6ef8e8042163589"
format_stamp: "Formatted at 2026-09-28 21:19:13 on dist-test-slave-rc9f"
server_key: "5da3c40115860fb7025eb17e6b3f944a"
server_key_iv: "dbaada302282de1055d8e5317af7754b"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:15.252907 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:15.275549 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:15.279522 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:15.279809 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:15.280623 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:15.281811 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:15.281924 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.282018 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:15.282078 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.292783 18151 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:34947
I20260928 21:19:15.295812 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:19:15.298691 18281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:34947 every 8 connection(s)
I20260928 21:19:15.299345 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18151
I20260928 21:19:15.299820 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.18.212:52773/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:34905
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:45321
--webserver_port=42863
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:15.322266 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:15.303861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:58827 (local address 127.5.18.254:45321)
0928 21:19:15.304809 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:15.304813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:15.305689 (+ 876us) server_negotiation.cc:407] Connection header received
0928 21:19:15.306690 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:15.306693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:15.306761 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:15.306842 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:15.308031 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.308557 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.309727 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.310045 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.310149 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.315030 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:15.315035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:15.315038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:15.315071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:15.316903 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.318273 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.318277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.318279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.318384 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.320786 (+ 2402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.320790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.320792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.321048 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:15.321143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:15.321913 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 21:19:15.322082 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825}
I20260928 21:19:15.324187 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:15.304501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.194:58827)
0928 21:19:15.305471 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:15.305532 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:15.306461 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:15.307061 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:15.307069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:15.307357 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:15.307878 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:15.307893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.308682 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.308685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:15.309620 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:15.309629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.310340 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.310344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:15.310550 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.311551 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:15.311572 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:15.314773 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:15.317077 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.317085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.317107 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.317557 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.318496 (+ 939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.318500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.318502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.320646 (+ 2144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.321200 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:15.321208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:15.321614 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 21:19:15.322750 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0928 21:19:15.323129 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":177,"threads_started":1}
I20260928 21:19:15.325711 18282 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:15.326092 18282 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.326841 18282 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
W20260928 21:19:15.328771 18282 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:45321 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:19:15.328980 18282 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.329229 18282 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:15.331712 18282 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.331935 18282 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
W20260928 21:19:15.332815 18282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:19:15.434823 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:15.435199 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:15.435261 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:15.441551 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:15.441701 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:15.441839 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:15.449821 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.18.212:52773/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:34905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=42863
--tserver_master_addrs=127.5.18.254:45321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:15.450953 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:15.452029 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:15.462170 18290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:15.462174 18293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:15.462188 18291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:15.467042 18284 server_base.cc:1034] running on GCE node
I20260928 21:19:15.467563 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:15.468400 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:15.470033 18284 hybrid_clock.cc:648] HybridClock initialized: now 1790630355470001 us; error 50 us; skew 500 ppm
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:15.473186 18284 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:15.474427 18284 webserver.cc:460] Webserver started at http://127.5.18.195:42863/ using document root <none> and password file <none>
I20260928 21:19:15.475203 18284 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:15.475385 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:15.498687 18284 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:19:15.501557 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.503006 18284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:19:15.503134 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "c7614629636045ebb97350ad45c9c6fc"
format_stamp: "Formatted at 2026-09-28 21:19:14 on dist-test-slave-rc9f"
server_key: "2a8fbc392361561dc0f7eef200ccc267"
server_key_iv: "3d9738dc73784e42ba8b971a19067134"
server_key_version: "kuduclusterkey@0"
I20260928 21:19:15.503710 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:15.519518 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:15.523238 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:15.523521 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:15.524255 18284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:15.525322 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:15.525462 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.525559 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:15.525653 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:15.535884 18284 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:34905
I20260928 21:19:15.536233 18414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:34905 every 8 connection(s)
I20260928 21:19:15.536932 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:19:15.541028 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18284
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:15.552551 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:15.542676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.5.18.254:45321)
0928 21:19:15.542890 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:15.542894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:15.542906 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:19:15.542944 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:15.542946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:15.542991 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:15.543936 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:15.544013 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.544719 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.545762 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.546120 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.546226 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.547957 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:15.547961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:15.547964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:15.547990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:15.550345 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.550869 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.550872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.550874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.550980 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.551337 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.551340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.551342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.551577 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:15.551785 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:15.552241 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 21:19:15.552364 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260928 21:19:15.554687 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:15.539143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56091 (local address 127.5.18.254:45321)
0928 21:19:15.539302 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:15.539307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:15.540172 (+ 865us) server_negotiation.cc:407] Connection header received
0928 21:19:15.541223 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:15.541227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:15.541282 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:15.541334 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:15.542976 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.543758 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.545454 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.545831 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.545940 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.548862 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:15.548866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:15.548868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:15.548927 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:15.550577 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.551145 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.551148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.551150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.551230 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:15.551868 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:15.551871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:15.551873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:15.552083 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:15.552169 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:15.554399 (+ 2230us) server_negotiation.cc:299] Negotiation successful
0928 21:19:15.554532 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:19:15.556419 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:15.539534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:45321 (local address 127.5.18.195:56091)
0928 21:19:15.539999 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:15.540052 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:15.540967 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:15.541751 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:15.541758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:15.542080 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:15.542813 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:15.542838 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.543912 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.543916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:15.545320 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:15.545332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:15.545990 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:15.545995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:15.546161 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:15.546858 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:15.546880 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:15.548676 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:15.550710 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.550718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.550738 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.551025 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.551548 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:15.551550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:15.551552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:15.551744 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:15.553639 (+ 1895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:15.553652 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:15.554085 (+ 433us) client_negotiation.cc:769] Sending connection context
0928 21:19:15.554859 (+ 774us) client_negotiation.cc:240] Negotiation successful
0928 21:19:15.555274 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260928 21:19:15.557638 18415 heartbeater.cc:347] Connected to a master server at 127.5.18.254:45321
I20260928 21:19:15.557883 18415 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:15.558609 18415 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:15.560089 17949 ts_manager.cc:194] Registered new tserver with Master: c7614629636045ebb97350ad45c9c6fc (127.5.18.195:34905)
I20260928 21:19:15.561725 17949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:56091
Sep 28 21:19:15 dist-test-slave-rc9f krb5kdc[16953](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:16.028720 18147 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:16.029350 18147 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:16.030839 17949 ts_manager.cc:194] Registered new tserver with Master: 1d58855e34b3406fb5b28136d90eeb87 (127.5.18.193:45863)
I20260928 21:19:16.032387 17949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:51993
Sep 28 21:19:16 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
I20260928 21:19:16.333091 18282 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:16.333443 18282 heartbeater.cc:511] Master 127.5.18.254:45321 requested a full tablet report, sending...
I20260928 21:19:16.334319 17949 ts_manager.cc:194] Registered new tserver with Master: 6c90c384beda4173a6ef8e8042163589 (127.5.18.194:34947)
I20260928 21:19:16.334944 17949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:58827
I20260928 21:19:16.336546 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:19:16.345420 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.338178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.5.18.254:45321)
0928 21:19:16.338475 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.338479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.338545 (+ 66us) server_negotiation.cc:407] Connection header received
0928 21:19:16.338664 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.338667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.338712 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.338792 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:16.339438 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.339974 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.340815 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.341044 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.341131 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.342487 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:16.342490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:16.342492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:16.342516 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:16.343912 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:16.344288 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:16.344291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:16.344293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:16.344369 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:16.344602 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:16.344605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:16.344607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:16.344789 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:16.344905 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.345121 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.345243 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":77,"threads_started":1}
I20260928 21:19:16.349432 17949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:19:16.354278 17949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:19:16.376487 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.366884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45863 (local address 127.0.0.1:44050)
0928 21:19:16.368805 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.368866 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:16.369380 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:16.370285 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:16.370292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:16.370318 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:16.370474 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.370482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.372481 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.372484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.374674 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.374685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.375990 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.375994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.376090 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.376114 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:19:16.376186 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:19:16.376323 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"thread_start_us":136,"threads_started":1}
I20260928 21:19:16.377168 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.369067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.5.18.194:34947)
0928 21:19:16.369370 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.369377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.370973 (+ 1596us) server_negotiation.cc:407] Connection header received
0928 21:19:16.371168 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.371174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.371384 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.371494 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:16.371883 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.373047 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.374580 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.375445 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.375520 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.375672 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.376927 (+ 1255us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.377031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
I20260928 21:19:16.377924 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.366978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.5.18.193:45863)
0928 21:19:16.369783 (+ 2805us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.369790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.369815 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:19:16.369894 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.369899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.370118 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.370271 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:16.370620 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.372343 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.374832 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.375665 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.375784 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.375925 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.377112 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.377215 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2697,"thread_start_us":102,"threads_started":1}
I20260928 21:19:16.377524 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.368682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:34947 (local address 127.0.0.1:48840)
0928 21:19:16.370886 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.370899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:16.371039 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:16.371652 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:16.371655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:16.371671 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:16.371772 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.371777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.373184 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.373187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.374446 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.374456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.375787 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.375790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.375897 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.375935 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:19:16.377314 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0928 21:19:16.377392 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":100,"threads_started":1}
I20260928 21:19:16.381832 18073 tablet_service.cc:1467] Processing CreateTablet for tablet 44dc764e39ad4a66921f42994b480b02 (DEFAULT_TABLE table=test-table [id=f361d871ec4b4a7da2e545996d8521e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:16.381834 18216 tablet_service.cc:1467] Processing CreateTablet for tablet 44dc764e39ad4a66921f42994b480b02 (DEFAULT_TABLE table=test-table [id=f361d871ec4b4a7da2e545996d8521e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:16.382975 18073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dc764e39ad4a66921f42994b480b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:16.382977 18216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dc764e39ad4a66921f42994b480b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:16.384363 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.366928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:34905 (local address 127.0.0.1:60224)
0928 21:19:16.369640 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.369653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:16.369730 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:16.376853 (+ 7123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:16.376856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:16.376877 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:16.376960 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.376966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.381636 (+ 4670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.381640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.382835 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.382847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.383979 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.383983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.384079 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.384107 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:19:16.384151 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 21:19:16.384220 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2649,"thread_start_us":109,"threads_started":1}
I20260928 21:19:16.385056 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.367808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.5.18.195:34905)
0928 21:19:16.376244 (+ 8436us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.376251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.376275 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:19:16.376347 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.376353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.376564 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.376706 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:16.380276 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.381464 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.383000 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.383846 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.384617 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.384775 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.384826 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.384917 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8318,"thread_start_us":87,"threads_started":1}
I20260928 21:19:16.388036 18336 tablet_service.cc:1467] Processing CreateTablet for tablet 44dc764e39ad4a66921f42994b480b02 (DEFAULT_TABLE table=test-table [id=f361d871ec4b4a7da2e545996d8521e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:16.389197 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44dc764e39ad4a66921f42994b480b02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:16.396080 18448 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Bootstrap starting.
I20260928 21:19:16.398571 18448 tablet_bootstrap.cc:654] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:16.399513 18448 log.cc:824] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:16.400372 18449 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Bootstrap starting.
I20260928 21:19:16.402637 18449 tablet_bootstrap.cc:654] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:16.403456 18449 log.cc:824] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:16.404096 18450 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Bootstrap starting.
I20260928 21:19:16.405788 18449 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: No bootstrap required, opened a new log
I20260928 21:19:16.406040 18449 ts_tablet_manager.cc:1397] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:19:16.406198 18450 tablet_bootstrap.cc:654] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:16.407020 18450 log.cc:824] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:16.409686 18448 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: No bootstrap required, opened a new log
I20260928 21:19:16.409896 18448 ts_tablet_manager.cc:1397] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20260928 21:19:16.410844 18449 raft_consensus.cc:348] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.412305 18449 raft_consensus.cc:374] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:16.412366 18449 raft_consensus.cc:729] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d58855e34b3406fb5b28136d90eeb87, State: Initialized, Role: FOLLOWER
I20260928 21:19:16.412916 18449 consensus_queue.cc:260] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.413905 18448 raft_consensus.cc:348] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.414142 18449 ts_tablet_manager.cc:1428] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 21:19:16.414254 18448 raft_consensus.cc:374] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:16.414353 18448 raft_consensus.cc:729] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c90c384beda4173a6ef8e8042163589, State: Initialized, Role: FOLLOWER
I20260928 21:19:16.414521 18147 heartbeater.cc:503] Master 127.5.18.254:45321 was elected leader, sending a full tablet report...
I20260928 21:19:16.414860 18448 consensus_queue.cc:260] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.415544 18282 heartbeater.cc:503] Master 127.5.18.254:45321 was elected leader, sending a full tablet report...
I20260928 21:19:16.415927 18448 ts_tablet_manager.cc:1428] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 21:19:16.418399 18450 tablet_bootstrap.cc:492] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: No bootstrap required, opened a new log
I20260928 21:19:16.418699 18450 ts_tablet_manager.cc:1397] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260928 21:19:16.423825 18450 raft_consensus.cc:348] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.424171 18450 raft_consensus.cc:374] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:16.424226 18450 raft_consensus.cc:729] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7614629636045ebb97350ad45c9c6fc, State: Initialized, Role: FOLLOWER
I20260928 21:19:16.424831 18450 consensus_queue.cc:260] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.426069 18450 ts_tablet_manager.cc:1428] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:19:16.426088 18415 heartbeater.cc:503] Master 127.5.18.254:45321 was elected leader, sending a full tablet report...
W20260928 21:19:16.488907 18148 tablet.cc:2369] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:19:16 dist-test-slave-rc9f krb5kdc[16953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630355, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for HTTP/127.5.18.212@KRBTEST.COM
W20260928 21:19:16.542990 18416 tablet.cc:2369] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:19:16.553035 18283 tablet.cc:2369] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:19:16.744292 18454 raft_consensus.cc:482] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:16.744521 18454 raft_consensus.cc:504] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.745827 18454 leader_election.cc:288] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7614629636045ebb97350ad45c9c6fc (127.5.18.195:34905), 6c90c384beda4173a6ef8e8042163589 (127.5.18.194:34947)
I20260928 21:19:16.760126 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.747144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:60183 (local address 127.5.18.195:34905)
0928 21:19:16.747418 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.747423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.751337 (+ 3914us) server_negotiation.cc:407] Connection header received
0928 21:19:16.751554 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.751558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.751612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.751657 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:16.752108 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.752913 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.756015 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.756625 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.756725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.756777 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.759807 (+ 3030us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.759930 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260928 21:19:16.760848 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.746236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:34905 (local address 127.5.18.193:60183)
0928 21:19:16.751223 (+ 4987us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.751242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:16.751430 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:16.751815 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:16.751817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:16.751837 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:16.751980 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.751987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.753063 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.753066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.754554 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.754566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.757141 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.757144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.757260 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.757282 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:19:16.760583 (+ 3301us) client_negotiation.cc:240] Negotiation successful
0928 21:19:16.760669 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4890,"thread_start_us":102,"threads_started":1}
I20260928 21:19:16.761727 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dc764e39ad4a66921f42994b480b02" candidate_uuid: "1d58855e34b3406fb5b28136d90eeb87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7614629636045ebb97350ad45c9c6fc" is_pre_election: true
I20260928 21:19:16.762094 18369 raft_consensus.cc:2454] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d58855e34b3406fb5b28136d90eeb87 in term 0.
I20260928 21:19:16.762773 18037 leader_election.cc:302] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d58855e34b3406fb5b28136d90eeb87, c7614629636045ebb97350ad45c9c6fc; no voters:
I20260928 21:19:16.763232 18454 raft_consensus.cc:2781] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:19:16.763347 18454 raft_consensus.cc:482] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:16.763391 18454 raft_consensus.cc:3037] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:16.764700 18454 raft_consensus.cc:504] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.765043 18454 leader_election.cc:288] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [CANDIDATE]: Term 1 election: Requested vote from peers c7614629636045ebb97350ad45c9c6fc (127.5.18.195:34905), 6c90c384beda4173a6ef8e8042163589 (127.5.18.194:34947)
I20260928 21:19:16.765460 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dc764e39ad4a66921f42994b480b02" candidate_uuid: "1d58855e34b3406fb5b28136d90eeb87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7614629636045ebb97350ad45c9c6fc"
I20260928 21:19:16.765604 18369 raft_consensus.cc:3037] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:16.766727 18369 raft_consensus.cc:2454] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d58855e34b3406fb5b28136d90eeb87 in term 1.
I20260928 21:19:16.767066 18037 leader_election.cc:302] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d58855e34b3406fb5b28136d90eeb87, c7614629636045ebb97350ad45c9c6fc; no voters:
I20260928 21:19:16.767784 18454 raft_consensus.cc:2781] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:16.767988 18454 raft_consensus.cc:686] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 1 LEADER]: Becoming Leader. State: Replica: 1d58855e34b3406fb5b28136d90eeb87, State: Running, Role: LEADER
I20260928 21:19:16.768337 18454 consensus_queue.cc:237] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } }
I20260928 21:19:16.771587 17949 catalog_manager.cc:5495] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d58855e34b3406fb5b28136d90eeb87 (127.5.18.193). New cstate: current_term: 1 leader_uuid: "1d58855e34b3406fb5b28136d90eeb87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d58855e34b3406fb5b28136d90eeb87" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 } health_report { overall_health: UNKNOWN } } }
I20260928 21:19:16.773204 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.747275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:39643 (local address 127.5.18.194:34947)
0928 21:19:16.747433 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.747437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.755840 (+ 8403us) server_negotiation.cc:407] Connection header received
0928 21:19:16.755979 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.755983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.756039 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.756101 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:16.757408 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.758233 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.767033 (+ 8800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.767597 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.767685 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.767728 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.772968 (+ 5240us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.773080 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:19:16.774111 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.746608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:34947 (local address 127.5.18.193:39643)
0928 21:19:16.755744 (+ 9136us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.755760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:16.755889 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:16.756844 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:16.756848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:16.756869 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:16.756976 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.756982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.758449 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.758452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.759647 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:16.759659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.772671 (+ 13012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.772675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:16.772793 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.772822 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 21:19:16.773542 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 21:19:16.773652 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9047,"thread_start_us":85,"threads_started":1}
I20260928 21:19:16.775002 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dc764e39ad4a66921f42994b480b02" candidate_uuid: "1d58855e34b3406fb5b28136d90eeb87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c90c384beda4173a6ef8e8042163589" is_pre_election: true
I20260928 21:19:16.774984 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44dc764e39ad4a66921f42994b480b02" candidate_uuid: "1d58855e34b3406fb5b28136d90eeb87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c90c384beda4173a6ef8e8042163589"
I20260928 21:19:16.775228 18236 raft_consensus.cc:3037] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:16.776279 18236 raft_consensus.cc:2454] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d58855e34b3406fb5b28136d90eeb87 in term 1.
I20260928 21:19:16.832113 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.828094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.5.18.193:45863)
0928 21:19:16.828207 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:16.828210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:16.828246 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:19:16.828418 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:16.828420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:16.828481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:16.828567 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:19:16.828828 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.829332 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.830559 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:16.830767 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:16.830840 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:16.831232 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:19:16.831664 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:19:16.831764 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:16.831876 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:19:16.831934 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:19:16.837945 18236 raft_consensus.cc:1261] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 1 FOLLOWER]: Refusing update from remote peer 1d58855e34b3406fb5b28136d90eeb87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:16.838016 18369 raft_consensus.cc:1261] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 1 FOLLOWER]: Refusing update from remote peer 1d58855e34b3406fb5b28136d90eeb87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:16.838649 18468 consensus_queue.cc:1035] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c90c384beda4173a6ef8e8042163589" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:16.838828 18454 consensus_queue.cc:1035] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7614629636045ebb97350ad45c9c6fc" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 34905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:16.855919 17948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:16.856130 17948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:16.858956 17948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04183ce1998242dfbb2f1f4c174885a8: Sending DeleteTablet for 3 replicas of tablet 44dc764e39ad4a66921f42994b480b02
I20260928 21:19:16.859822 18073 tablet_service.cc:1514] Processing DeleteTablet for tablet 44dc764e39ad4a66921f42994b480b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:16 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:44050
I20260928 21:19:16.859931 18336 tablet_service.cc:1514] Processing DeleteTablet for tablet 44dc764e39ad4a66921f42994b480b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:16 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:60224
I20260928 21:19:16.860428 18480 tablet_replica.cc:330] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87: stopping tablet replica
I20260928 21:19:16.860394 18216 tablet_service.cc:1514] Processing DeleteTablet for tablet 44dc764e39ad4a66921f42994b480b02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:16 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:48840
I20260928 21:19:16.860749 18480 raft_consensus.cc:2229] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:19:16.860875 18482 tablet_replica.cc:330] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: stopping tablet replica
I20260928 21:19:16.861101 18481 tablet_replica.cc:330] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: stopping tablet replica
I20260928 21:19:16.861178 18480 raft_consensus.cc:2258] T 44dc764e39ad4a66921f42994b480b02 P 1d58855e34b3406fb5b28136d90eeb87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:16.861284 18482 raft_consensus.cc:2229] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:16.861514 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17987
I20260928 21:19:16.861529 18481 raft_consensus.cc:2229] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:16.862568 18481 raft_consensus.cc:2258] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:16.863266 18482 raft_consensus.cc:2258] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:16.864236 18481 ts_tablet_manager.cc:1905] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:16.864511 18482 ts_tablet_manager.cc:1905] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:16.866920 18482 ts_tablet_manager.cc:1918] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:16.867014 18482 log.cc:1196] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/44dc764e39ad4a66921f42994b480b02
I20260928 21:19:16.867082 18481 ts_tablet_manager.cc:1918] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:16.867218 18481 log.cc:1196] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/44dc764e39ad4a66921f42994b480b02
I20260928 21:19:16.867378 18482 ts_tablet_manager.cc:1939] T 44dc764e39ad4a66921f42994b480b02 P 6c90c384beda4173a6ef8e8042163589: Deleting consensus metadata
I20260928 21:19:16.867520 18481 ts_tablet_manager.cc:1939] T 44dc764e39ad4a66921f42994b480b02 P c7614629636045ebb97350ad45c9c6fc: Deleting consensus metadata
I20260928 21:19:16.868692 17936 catalog_manager.cc:4841] TS c7614629636045ebb97350ad45c9c6fc (127.5.18.195:34905): tablet 44dc764e39ad4a66921f42994b480b02 (table test-table [id=f361d871ec4b4a7da2e545996d8521e3]) successfully deleted
I20260928 21:19:16.868920 17936 catalog_manager.cc:4841] TS 6c90c384beda4173a6ef8e8042163589 (127.5.18.194:34947): tablet 44dc764e39ad4a66921f42994b480b02 (table test-table [id=f361d871ec4b4a7da2e545996d8521e3]) successfully deleted
W20260928 21:19:16.869259 17934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:45863 (error 108)
I20260928 21:19:16.869393 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18151
I20260928 21:19:16.870836 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:16.870489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45863 (local address 127.0.0.1:44070)
0928 21:19:16.870671 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:16.870743 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 21:19:16.871012 17934 catalog_manager.cc:4584] TS 1d58855e34b3406fb5b28136d90eeb87 (127.5.18.193:45863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44dc764e39ad4a66921f42994b480b02: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45863: connect: Connection refused (error 111)
I20260928 21:19:16.874403 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18284
I20260928 21:19:16.879387 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 17916
I20260928 21:19:16.885210 5195 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:19:17.220568 5195 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:19:17.220685 5195 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:19:17.555918 5195 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:19:17.556 UTC [16983] LOG: received smart shutdown request
2026-09-28 21:19:17.560 UTC [16983] LOG: background worker "logical replication launcher" (PID 16992) exited with exit code 1
2026-09-28 21:19:17.560 UTC [16987] LOG: shutting down
2026-09-28 21:19:17.728 UTC [16983] LOG: database system is shut down
2026-09-28T21:19:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48510 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:19:17 dist-test-slave-rc9f krb5kdc[18492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:19:17 dist-test-slave-rc9f krb5kdc[18492](info): set up 2 sockets
Sep 28 21:19:17 dist-test-slave-rc9f krb5kdc[18492](info): commencing operation
krb5kdc: starting...
W20260928 21:19:18.793406 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:19:18 dist-test-slave-rc9f krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:19:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:19:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:19:18.916033 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42403
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:19.029088 18505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:19.029402 18505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:19.029464 18505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:19.033524 18505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:19.033618 18505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:19.033648 18505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:19.033668 18505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:19.033689 18505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:19.039839 18505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:19.041232 18505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:19.042526 18505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:19.051213 18511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.051236 18513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.051229 18510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:19.051287 18505 server_base.cc:1034] running on GCE node
I20260928 21:19:19.052130 18505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:19.053311 18505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:19.054486 18505 hybrid_clock.cc:648] HybridClock initialized: now 1790630359054464 us; error 42 us; skew 500 ppm
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:19.061031 18505 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:19.062179 18505 webserver.cc:460] Webserver started at http://127.5.18.254:39951/ using document root <none> and password file <none>
I20260928 21:19:19.062800 18505 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:19.062875 18505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:19.063094 18505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:19.065270 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "5522e9b72d2e06a05d68144ae3c13ac0"
server_key_iv: "f9935e95ae8ca9841a5b610894b1372e"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.065881 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "5522e9b72d2e06a05d68144ae3c13ac0"
server_key_iv: "f9935e95ae8ca9841a5b610894b1372e"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.069736 18505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:19.072337 18520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:19.073730 18505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:19:19.073834 18505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "5522e9b72d2e06a05d68144ae3c13ac0"
server_key_iv: "f9935e95ae8ca9841a5b610894b1372e"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.073933 18505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:19.087404 18505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:19.090306 18505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:19.090469 18505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:19.097464 18505 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:42403
I20260928 21:19:19.097491 18572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:42403 every 8 connection(s)
I20260928 21:19:19.098513 18505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:19.101457 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:19.102267 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18505
I20260928 21:19:19.102406 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:19:19.102721 5195 external_mini_cluster.cc:1428] Setting key 7f08c39d07042c8a77423e60c9eb10ea
I20260928 21:19:19.106106 18573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.18.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.253@KRBTEST.COM" created.
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:19.121558 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.108428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42403 (local address 127.0.0.1:36356)
0928 21:19:19.109012 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.109080 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.110222 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.110874 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.110879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.111228 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.111491 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.111505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.112660 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.112664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.113547 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.113557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.114467 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.114472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.114581 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.115154 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.115175 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.116762 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.118645 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.118657 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.118669 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.118972 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.119249 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.119252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.119254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.119402 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.120002 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.120011 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.120183 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.120390 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.120807 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":87,"thread_start_us":126,"threads_started":1}
I20260928 21:19:19.121591 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.109198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.5.18.254:42403)
0928 21:19:19.109461 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.109470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.109507 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:19:19.110505 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.110510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.110706 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.110860 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.111622 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.112523 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.113718 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.114293 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.114463 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.116917 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.116924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.116935 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.116957 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.118537 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.119065 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.119069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.119070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.119171 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.119492 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.119494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.119495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.119878 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.119989 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.120430 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.120710 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260928 21:19:19.124198 18573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:19:19.124940 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.124672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:45978)
0928 21:19:19.124793 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.124871 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:19:19.125166 18521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
I20260928 21:19:19.126032 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.125825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:45986)
0928 21:19:19.125930 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.125976 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 21:19:19.126168 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:41555: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
I20260928 21:19:19.129042 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:41555
--webserver_interface=127.5.18.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:19.175458 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } attempt: 1
I20260928 21:19:19.176357 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.176028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:45994)
0928 21:19:19.176193 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.176268 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:19:19.177057 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.176846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46002)
0928 21:19:19.176939 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.176991 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:19:19.177179 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:41555: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
W20260928 21:19:19.241211 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:19.241501 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:19.241541 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:19:19.245342 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } attempt: 2
W20260928 21:19:19.245674 18580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:19.245726 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:19.245746 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:19.245775 18580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:19.245800 18580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:19.246238 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.245926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46008)
0928 21:19:19.246079 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.246153 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:19:19.246884 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.246674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46010)
0928 21:19:19.246766 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.246815 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:19:19.247025 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:41555: Network error: Client connection negotiation failed: client connection to 127.5.18.253:41555: connect: Connection refused (error 111)
I20260928 21:19:19.251721 18580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.18.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:19.252796 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:19.253846 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:19.262018 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.262037 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.262018 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:19.262269 18580 server_base.cc:1034] running on GCE node
I20260928 21:19:19.262971 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:19.264081 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:19.265241 18580 hybrid_clock.cc:648] HybridClock initialized: now 1790630359265196 us; error 60 us; skew 500 ppm
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:19.267976 18580 init.cc:377] Logged in from keytab as kudu/127.5.18.253@KRBTEST.COM (short username kudu)
I20260928 21:19:19.268899 18580 webserver.cc:460] Webserver started at http://127.5.18.253:46715/ using document root <none> and password file <none>
I20260928 21:19:19.269438 18580 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:19.269502 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:19.269733 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:19.271791 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data/instance:
uuid: "dfb9490ff4164fbea2fefbae0a59e6d2"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "0e442a771eb87df9fa86be87742e3452"
server_key_iv: "666957245f49ac7320a0bf61eba6490f"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.272330 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal/instance:
uuid: "dfb9490ff4164fbea2fefbae0a59e6d2"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "0e442a771eb87df9fa86be87742e3452"
server_key_iv: "666957245f49ac7320a0bf61eba6490f"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.276134 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:19:19.278607 18595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:19.279836 18580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:19.279976 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "dfb9490ff4164fbea2fefbae0a59e6d2"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "0e442a771eb87df9fa86be87742e3452"
server_key_iv: "666957245f49ac7320a0bf61eba6490f"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.280076 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:19.292711 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:19.295677 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:19.295876 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:19.302690 18580 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.253:41555
I20260928 21:19:19.302711 18647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.253:41555 every 8 connection(s)
I20260928 21:19:19.303709 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
I20260928 21:19:19.305193 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18580
I20260928 21:19:19.305320 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-1/wal/instance
I20260928 21:19:19.305680 5195 external_mini_cluster.cc:1428] Setting key 246e005d349257d3d0ac94ad5e041e78
I20260928 21:19:19.307080 18648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:19.311125 18648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.18.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.252@KRBTEST.COM" created.
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:19.326061 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.313018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.5.18.254:42403)
0928 21:19:19.313160 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.313165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.314216 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 21:19:19.315255 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.315264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.315324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.315428 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.316889 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.317646 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.319118 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.319507 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.319616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.322533 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.322536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.322538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.322567 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.323937 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.324461 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.324464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.324466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.324544 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.324879 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.324882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.324884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.325073 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.325159 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.325750 (+ 591us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.325895 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:19:19.326921 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.313431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42403 (local address 127.0.0.1:36362)
0928 21:19:19.314059 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.314104 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.314966 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.315684 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.315693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.316160 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.316714 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.316728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.317790 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.317794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.318983 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.318993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.319672 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.319675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.319897 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.320601 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.320624 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.322343 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.324059 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.324066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.324084 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.324363 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.324628 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.324631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.324633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.324792 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.325176 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.325183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.325476 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.325687 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.326054 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":157,"threads_started":1}
I20260928 21:19:19.328586 18648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
I20260928 21:19:19.333241 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:44285
--webserver_interface=127.5.18.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:19:19.338202 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } attempt: 3
I20260928 21:19:19.338398 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.329154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46016)
0928 21:19:19.329287 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.329299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.329378 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.330005 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.330007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.330162 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.330318 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.330325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.331359 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.331362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.332002 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.332035 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.332857 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.332860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.332957 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.333516 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.333530 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.334960 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.336700 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.336703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.336706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.336960 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.337209 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.337212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.337214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.337316 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.337811 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.337814 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.337928 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.338045 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.338219 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 21:19:19.338649 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.329267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.5.18.253:41555)
0928 21:19:19.329640 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.329651 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.329671 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:19:19.329710 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.329714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.329882 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.329999 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.330442 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.331245 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.332175 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.332724 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.332831 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.335093 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.335103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.335116 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.335137 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.336583 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.337058 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.337062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.337064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.337132 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.337392 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.337394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.337395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.337723 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.337825 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.338110 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.338316 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":259,"thread_start_us":153,"threads_started":1}
I20260928 21:19:19.341015 18648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:19:19.341805 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.341471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49406)
0928 21:19:19.341634 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.341722 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:19:19.342002 18596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
I20260928 21:19:19.343225 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.342992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49420)
0928 21:19:19.343104 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.343156 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
W20260928 21:19:19.344028 18648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
I20260928 21:19:19.347859 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.338670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46032)
0928 21:19:19.338833 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.338845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.338912 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.339445 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.339448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.339616 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.340101 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.340110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.340848 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.340851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.341618 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.341629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.342195 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.342199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.342301 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.342767 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.342782 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.344200 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.346482 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.346485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.346486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.346675 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.346928 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.346930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.346931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.347002 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.347416 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.347419 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.347519 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.347613 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.347737 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 21:19:19.347941 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.339140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.5.18.253:41555)
0928 21:19:19.339229 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.339232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.339243 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:19:19.339270 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.339273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.339309 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.340194 (+ 885us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.340261 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.340747 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.341740 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.342062 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.342187 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.344322 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.344325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.344328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.344352 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.346361 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.346759 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.346762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.346765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.346846 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.347072 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.347075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.347077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.347298 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.347391 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.347670 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.347789 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:19:19.348753 18573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:19:19.349459 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.349214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49428)
0928 21:19:19.349335 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.349389 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:19:19.350126 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.349892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49438)
0928 21:19:19.350033 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.350075 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 21:19:19.350255 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
I20260928 21:19:19.381430 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 1
I20260928 21:19:19.382267 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.381984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49442)
0928 21:19:19.382129 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.382190 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:19:19.382897 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.382710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49450)
0928 21:19:19.382804 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.382847 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:19:19.383046 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
I20260928 21:19:19.393359 18648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 1
I20260928 21:19:19.394340 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.394001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49460)
0928 21:19:19.394180 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.394256 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 21:19:19.395085 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.394863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49462)
0928 21:19:19.394970 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.395021 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 21:19:19.395205 18648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
W20260928 21:19:19.446205 18654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:19.446506 18654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:19.446563 18654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:19.450649 18654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:19.450718 18654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:19.450744 18654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:19.450767 18654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:19.450796 18654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:19.456359 18654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42403,127.5.18.253:41555,127.5.18.252:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:44285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.18.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:19.457521 18654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:19.458202 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 2
I20260928 21:19:19.458797 18654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:19:19.459017 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.458725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49478)
0928 21:19:19.458872 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.458939 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:19:19.459733 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.459541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49492)
0928 21:19:19.459629 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.459675 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:19:19.459851 18573 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
I20260928 21:19:19.463308 18648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 2
I20260928 21:19:19.463968 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.463730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49506)
0928 21:19:19.463856 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.463907 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:19:19.464521 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.464350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49514)
0928 21:19:19.464434 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.464473 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:19:19.464637 18648 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:44285: Network error: Client connection negotiation failed: client connection to 127.5.18.252:44285: connect: Connection refused (error 111)
W20260928 21:19:19.467471 18660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.467502 18663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:19.467558 18661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:19.467832 18654 server_base.cc:1034] running on GCE node
I20260928 21:19:19.468407 18654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:19.469558 18654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:19.470764 18654 hybrid_clock.cc:648] HybridClock initialized: now 1790630359470733 us; error 44 us; skew 500 ppm
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:19.473423 18654 init.cc:377] Logged in from keytab as kudu/127.5.18.252@KRBTEST.COM (short username kudu)
I20260928 21:19:19.474366 18654 webserver.cc:460] Webserver started at http://127.5.18.252:41363/ using document root <none> and password file <none>
I20260928 21:19:19.475072 18654 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:19.475157 18654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:19.475411 18654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:19.477485 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data/instance:
uuid: "6ff82da42b53436ebdf0d5c378fa923e"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "57a01eb6dbc1be273f2a48404e0dc7ee"
server_key_iv: "3ff349739fdfc9fa33b1ea2a19adf2a0"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.478034 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal/instance:
uuid: "6ff82da42b53436ebdf0d5c378fa923e"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "57a01eb6dbc1be273f2a48404e0dc7ee"
server_key_iv: "3ff349739fdfc9fa33b1ea2a19adf2a0"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.481803 18654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:19.484208 18670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:19.485432 18654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:19.485539 18654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "6ff82da42b53436ebdf0d5c378fa923e"
format_stamp: "Formatted at 2026-09-28 21:19:19 on dist-test-slave-rc9f"
server_key: "57a01eb6dbc1be273f2a48404e0dc7ee"
server_key_iv: "3ff349739fdfc9fa33b1ea2a19adf2a0"
server_key_version: "encryptionkey@0"
I20260928 21:19:19.485677 18654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:19.498162 18654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:19.500936 18654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:19.501118 18654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:19.507910 18654 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.252:44285
I20260928 21:19:19.507946 18722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.252:44285 every 8 connection(s)
I20260928 21:19:19.508903 18654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
I20260928 21:19:19.509188 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18654
I20260928 21:19:19.509322 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790630206151656-5195-0/minicluster-data/master-2/wal/instance
I20260928 21:19:19.509688 5195 external_mini_cluster.cc:1428] Setting key 7d8a349cf1eb940d1500626a6427edc4
I20260928 21:19:19.512346 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:19.517097 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:19.524801 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.511087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.5.18.252:44285)
0928 21:19:19.511523 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.511542 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.511584 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:19:19.512434 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.512460 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.512885 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.513267 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.514003 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.515507 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.516414 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.517084 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.517247 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.519797 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.519803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.519822 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.519856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.521763 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.522262 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.522267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.522272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.522467 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.522730 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.522733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.522734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.523194 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.523384 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.523675 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.523968 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260928 21:19:19.528302 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.519049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:42403 (local address 127.0.0.1:36376)
0928 21:19:19.519488 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.519541 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.519670 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.519950 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.519954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.520299 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.520440 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.520445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.521430 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.521433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.522262 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.522273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.522769 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.522772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.522861 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.523373 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.523385 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.524657 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.526356 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.526362 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.526368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.526633 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.526931 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.526935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.526937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.527032 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.527632 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.527639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.527783 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.527934 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.528136 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":768,"thread_start_us":166,"threads_started":1}
I20260928 21:19:19.528329 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.519122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.5.18.254:42403)
0928 21:19:19.519266 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.519271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.519636 (+ 365us) server_negotiation.cc:407] Connection header received
0928 21:19:19.519759 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.519762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.519826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.519909 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.520565 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.521314 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.522364 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.522666 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.522758 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.524796 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.524801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.524803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.524834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.526258 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.526725 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.526729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.526730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.526838 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.527122 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.527125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.527126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.527460 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.527573 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.528047 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.528170 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:19:19.529531 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:19:19.537460 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.527366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.5.18.254:42403)
0928 21:19:19.527762 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.527771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.527783 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:19:19.527817 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.527819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.527862 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.527961 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.529082 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.529871 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.530916 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.531169 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.531243 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.534051 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.534055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.534059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.534091 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.535816 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536226 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.536229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.536231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.536318 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536573 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.536576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.536578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.536774 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.536976 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.537148 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.537267 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":115,"threads_started":1}
I20260928 21:19:19.537555 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.530058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:41555 (local address 127.0.0.1:46034)
0928 21:19:19.530230 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.530244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.530313 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.530521 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.530523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.530655 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.530749 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.530753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.531530 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.531533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.532163 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.532174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.532758 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.532761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.532856 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.533291 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.533305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.534630 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.536169 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.536175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.536375 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.536632 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.536637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.536718 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.537063 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.537066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.537159 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.537240 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.537382 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 21:19:19.537856 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.530180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.5.18.253:41555)
0928 21:19:19.530322 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.530327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.530344 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:19:19.530387 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.530390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.530435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.530517 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.530849 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.531422 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.532294 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.532654 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.532782 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.534771 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.534774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.534776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.534802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.536063 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536475 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.536479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.536480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.536551 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.536794 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.536797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.536799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.536977 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.537059 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.537300 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.537403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:19:19.538389 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:19:19.546816 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.538950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49546)
0928 21:19:19.539047 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.539059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.539116 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.539369 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.539371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.539496 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.539571 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.539576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.540803 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.540806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.541483 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.541493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.542086 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.542089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.542205 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.542585 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.542595 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.543823 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.545346 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.545349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.545351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.545530 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.545800 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.545803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.545805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.545874 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.546262 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.546264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.546364 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.546474 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.546612 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":20}
I20260928 21:19:19.546819 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.539085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.5.18.252:44285)
0928 21:19:19.539193 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.539197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.539209 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:19:19.539239 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.539242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.539281 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.539361 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.539668 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.540679 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.541646 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.541982 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.542077 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.543974 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.543977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.543979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.544006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.545253 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.545648 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.545651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.545653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.545728 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.545945 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.545948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.545950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.546163 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.546261 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.546531 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.546685 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":36}
I20260928 21:19:19.547783 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.538720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.5.18.253:41555)
0928 21:19:19.538823 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.538827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.538839 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:19:19.538963 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.538966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.539009 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.539084 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.539755 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.540236 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.541140 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.541508 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.541642 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.544504 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.544507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.544509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.544531 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.545928 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.546318 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.546321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.546323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.546402 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.546665 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.546668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.546670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.546870 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.547209 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.547532 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.547642 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:19:19.550474 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: Bootstrap starting.
I20260928 21:19:19.551177 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:19:19.553474 18723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:19.554481 18723 log.cc:824] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:19.555773 18648 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 3
I20260928 21:19:19.557940 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: No bootstrap required, opened a new log
I20260928 21:19:19.558990 18573 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } attempt: 3
I20260928 21:19:19.563035 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.553401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.5.18.254:42403)
0928 21:19:19.553539 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.553544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.553889 (+ 345us) server_negotiation.cc:407] Connection header received
0928 21:19:19.553975 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.553978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.554039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.554121 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.554991 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.555622 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.557001 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.557343 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.557446 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.559276 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.559279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.559282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.559313 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.560901 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.561559 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.561590 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.561592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.561682 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.561981 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.561985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.561987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.562194 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.562329 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.562740 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.562873 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:19:19.563611 18723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.563941 18723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:19.564007 18723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff82da42b53436ebdf0d5c378fa923e, State: Initialized, Role: FOLLOWER
I20260928 21:19:19.564558 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.553813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.5.18.253:41555)
0928 21:19:19.553923 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.553927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.554509 (+ 582us) server_negotiation.cc:407] Connection header received
0928 21:19:19.554671 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.554674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.554724 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.554768 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.555637 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.556459 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.557618 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.558301 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.558395 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.560469 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.560472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.560475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.560501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.562231 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.562938 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.562942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.562944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.563250 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.563573 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.563576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.563578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.563797 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.563933 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.564278 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.564399 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:19:19.564632 18723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.566205 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.554360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.5.18.252:44285)
0928 21:19:19.554469 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.554473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.555944 (+ 1471us) server_negotiation.cc:407] Connection header received
0928 21:19:19.556137 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.556140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.556187 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.556234 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.558179 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.558896 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.560030 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.560383 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.560452 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.562280 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.562284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.562287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.562312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.563833 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.564532 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.564536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.564537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.564623 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.565182 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.565185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.565187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.565443 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.565559 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.565940 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.566061 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:19:19.566258 18723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:19.568889 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.569015 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
W20260928 21:19:19.572156 18750 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:19:19.572238 18750 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 21:19:19 dist-test-slave-rc9f krb5kdc[18492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630359, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:19:19.570096 18654 master_runner.cc:186] Error getting master registration for 127.5.18.254:42403: OK, instance_id { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" instance_seqno: 1790630359095487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:19:19.573836 18654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:19:19.574330 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.559399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49578)
0928 21:19:19.559658 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.559671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.559745 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.561197 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.561200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.561360 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.561465 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.561472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.566009 (+ 4537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.566012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.566756 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.566770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.567518 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.567522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.567618 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.568158 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.568173 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.569823 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.572522 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.572525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.572527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.572777 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.573071 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.573074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.573076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.573181 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.573792 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.573795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.573900 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.574025 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.574176 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 21:19:19.574366 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.559876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.5.18.252:44285)
0928 21:19:19.560210 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.560218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.560232 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:19:19.561023 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.561026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.561073 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.564925 (+ 3852us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.565068 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.565861 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.566905 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.567330 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.567863 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.570019 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.570023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.570026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.570054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.571617 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.572859 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.572862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.572864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.572971 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.573285 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.573288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.573290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.573655 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.573746 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.574099 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.574226 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260928 21:19:19.577584 18573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: Bootstrap starting.
I20260928 21:19:19.579079 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.556186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:44285 (local address 127.0.0.1:49576)
0928 21:19:19.556441 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:19.556453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:19.556527 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:19.557192 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:19.557195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:19.557390 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:19.557491 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.557498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.567252 (+ 9754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.567256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.568311 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:19.568325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.569471 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.569475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:19.569594 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.570115 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:19.570129 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:19.573990 (+ 3861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:19.577244 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.577248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.577250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.577553 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.577901 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:19.577905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:19.577907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:19.578015 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:19.578542 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:19.578545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:19.578655 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 21:19:19.578761 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:19:19.578917 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260928 21:19:19.579159 18737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:19.556648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49576 (local address 127.5.18.252:44285)
0928 21:19:19.556961 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:19.556969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:19.556983 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:19:19.557017 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:19.557019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:19.557058 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:19.566137 (+ 9079us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:19.566238 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.567130 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.568463 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:19.568801 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:19.571109 (+ 2308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:19.574654 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:19.574658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:19.574661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:19.574686 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:19.577100 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.577668 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.577672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.577674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.577783 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:19.578129 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:19.578132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:19.578134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:19.578380 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:19.578546 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:19.578854 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 21:19:19.578981 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":88,"threads_started":1}
I20260928 21:19:19.580452 18573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:19.581382 18573 log.cc:824] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:19.582190 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: Bootstrap starting.
I20260928 21:19:19.584717 18648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:19.585616 18648 log.cc:824] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:19.587659 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: No bootstrap required, opened a new log
I20260928 21:19:19.591151 18573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: No bootstrap required, opened a new log
I20260928 21:19:19.591028 18648 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.591279 18648 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:19.591351 18648 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb9490ff4164fbea2fefbae0a59e6d2, State: Initialized, Role: FOLLOWER
I20260928 21:19:19.591996 18648 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.593211 18648 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:19.595350 18754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.595508 18754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:19:19.596510 18573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.596815 18573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:19.596913 18573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e7c22a2d254e36baf1e53a7ffa6e5a, State: Initialized, Role: FOLLOWER
I20260928 21:19:19.597457 18573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
W20260928 21:19:19.598654 18765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:19:19.598731 18765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:19:19.599027 18573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:19.596529 18580 master_runner.cc:186] Error getting master registration for 127.5.18.254:42403: OK, instance_id { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" instance_seqno: 1790630359095487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:19:19.599687 18580 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:19:19.601442 18767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.601619 18767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:19:19.605520 18778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:19:19.605638 18778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:19:19.608083 18505 master_runner.cc:263] No leader master found from master 59e7c22a2d254e36baf1e53a7ffa6e5a
I20260928 21:19:19.608162 18505 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:19:19.752620 18676 tablet.cc:2369] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:19:19.764209 18754 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:19.764427 18754 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.765318 18754 leader_election.cc:288] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59e7c22a2d254e36baf1e53a7ffa6e5a (127.5.18.254:42403), 6ff82da42b53436ebdf0d5c378fa923e (127.5.18.252:44285)
I20260928 21:19:19.765846 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff82da42b53436ebdf0d5c378fa923e" is_pre_election: true
I20260928 21:19:19.766165 18697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb9490ff4164fbea2fefbae0a59e6d2 in term 0.
I20260928 21:19:19.766566 18596 leader_election.cc:302] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff82da42b53436ebdf0d5c378fa923e, dfb9490ff4164fbea2fefbae0a59e6d2; no voters:
I20260928 21:19:19.766690 18547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" is_pre_election: true
I20260928 21:19:19.766978 18547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb9490ff4164fbea2fefbae0a59e6d2 in term 0.
I20260928 21:19:19.767205 18754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:19:19.767277 18754 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:19.767336 18754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:19.768412 18754 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.768743 18754 leader_election.cc:288] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [CANDIDATE]: Term 1 election: Requested vote from peers 59e7c22a2d254e36baf1e53a7ffa6e5a (127.5.18.254:42403), 6ff82da42b53436ebdf0d5c378fa923e (127.5.18.252:44285)
I20260928 21:19:19.769289 18697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff82da42b53436ebdf0d5c378fa923e"
I20260928 21:19:19.769420 18697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:19.770434 18697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb9490ff4164fbea2fefbae0a59e6d2 in term 1.
I20260928 21:19:19.770721 18596 leader_election.cc:302] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ff82da42b53436ebdf0d5c378fa923e, dfb9490ff4164fbea2fefbae0a59e6d2; no voters:
I20260928 21:19:19.770889 18754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:19.771122 18754 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [term 1 LEADER]: Becoming Leader. State: Replica: dfb9490ff4164fbea2fefbae0a59e6d2, State: Running, Role: LEADER
I20260928 21:19:19.771456 18754 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } }
I20260928 21:19:19.773046 18547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a"
I20260928 21:19:19.773190 18547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:19.774511 18547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb9490ff4164fbea2fefbae0a59e6d2 in term 1.
I20260928 21:19:19.775135 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb9490ff4164fbea2fefbae0a59e6d2. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.775264 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:19.775619 18782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:19.776439 18782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:19.781865 18547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [term 1 FOLLOWER]: Refusing update from remote peer dfb9490ff4164fbea2fefbae0a59e6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:19.782181 18697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [term 1 FOLLOWER]: Refusing update from remote peer dfb9490ff4164fbea2fefbae0a59e6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:19.782891 18754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:19.783205 18780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:19.787429 18767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb9490ff4164fbea2fefbae0a59e6d2. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.787560 18767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:19:19.787483 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfb9490ff4164fbea2fefbae0a59e6d2. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.787611 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:19:19.790206 18754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.790338 18754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:19.790913 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.791038 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:19:19.791270 18784 mvcc.cc:205] Tried to move back new op lower bound from 7334421953657495552 to 7334421953626525696. Current Snapshot: MvccSnapshot[applied={T|T < 7334421953657495552}]
I20260928 21:19:19.792104 18782 catalog_manager.cc:1347] Generated new cluster ID: 30049c08f0de43cdb59134e78f3bd124
I20260928 21:19:19.792173 18782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:19.794507 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.794638 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:19.795758 18785 mvcc.cc:205] Tried to move back new op lower bound from 7334421953657495552 to 7334421953626525696. Current Snapshot: MvccSnapshot[applied={T|T < 7334421953657495552}]
I20260928 21:19:19.796033 18767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e7c22a2d254e36baf1e53a7ffa6e5a" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42403 } } peers { permanent_uuid: "dfb9490ff4164fbea2fefbae0a59e6d2" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 41555 } } peers { permanent_uuid: "6ff82da42b53436ebdf0d5c378fa923e" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 44285 } } }
I20260928 21:19:19.796137 18767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:19:19.806900 18782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:19:19.808244 18782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:19.816680 18782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfb9490ff4164fbea2fefbae0a59e6d2: Generated new TSK 0
I20260928 21:19:19.817982 18782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:19:20.573513 18750 catalog_manager.cc:1259] Loaded cluster ID: 30049c08f0de43cdb59134e78f3bd124
I20260928 21:19:20.573632 18750 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: loading cluster ID for follower catalog manager: success
I20260928 21:19:20.575333 18750 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: acquiring CA information for follower catalog manager: success
I20260928 21:19:20.576524 18750 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ff82da42b53436ebdf0d5c378fa923e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:19:20.607036 18778 catalog_manager.cc:1259] Loaded cluster ID: 30049c08f0de43cdb59134e78f3bd124
I20260928 21:19:20.607126 18778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: loading cluster ID for follower catalog manager: success
I20260928 21:19:20.608805 18778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: acquiring CA information for follower catalog manager: success
I20260928 21:19:20.610175 18778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 59e7c22a2d254e36baf1e53a7ffa6e5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:19:20.878083 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18505
I20260928 21:19:20.883663 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18580
I20260928 21:19:20.888906 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18654
2026-09-28T21:19:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3048 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:19:20 dist-test-slave-rc9f krb5kdc[18797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:19:20 dist-test-slave-rc9f krb5kdc[18797](info): set up 2 sockets
Sep 28 21:19:20 dist-test-slave-rc9f krb5kdc[18797](info): commencing operation
krb5kdc: starting...
W20260928 21:19:21.845557 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:19:21 dist-test-slave-rc9f krb5kdc[18797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:19:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:19:21.964450 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38791
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:22.073705 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:22.074043 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:22.074105 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:22.078289 18810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:22.078370 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:22.078397 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:22.078418 18810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:22.078439 18810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:22.084825 18810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:22.086024 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:22.087092 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:22.095850 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:22.095876 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:22.096029 18810 server_base.cc:1034] running on GCE node
W20260928 21:19:22.095867 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:22.096781 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:22.098052 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:22.099221 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790630362099206 us; error 35 us; skew 500 ppm
Sep 28 21:19:22 dist-test-slave-rc9f krb5kdc[18797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630362, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:22.102077 18810 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:22.103000 18810 webserver.cc:460] Webserver started at http://127.5.18.254:42321/ using document root <none> and password file <none>
I20260928 21:19:22.103557 18810 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:22.103617 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:22.103793 18810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:22.105940 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "7e3604480e534a56b0ac9876d99a6941"
format_stamp: "Formatted at 2026-09-28 21:19:22 on dist-test-slave-rc9f"
server_key: "c0d3818decdee5e03301f89c6be508f7"
server_key_iv: "6b47dac838a3c3aaf013728a42ef5169"
server_key_version: "encryptionkey@0"
I20260928 21:19:22.106460 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "7e3604480e534a56b0ac9876d99a6941"
format_stamp: "Formatted at 2026-09-28 21:19:22 on dist-test-slave-rc9f"
server_key: "c0d3818decdee5e03301f89c6be508f7"
server_key_iv: "6b47dac838a3c3aaf013728a42ef5169"
server_key_version: "encryptionkey@0"
I20260928 21:19:22.110175 18810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:19:22.112582 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:22.113885 18810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:22.114017 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "7e3604480e534a56b0ac9876d99a6941"
format_stamp: "Formatted at 2026-09-28 21:19:22 on dist-test-slave-rc9f"
server_key: "c0d3818decdee5e03301f89c6be508f7"
server_key_iv: "6b47dac838a3c3aaf013728a42ef5169"
server_key_version: "encryptionkey@0"
I20260928 21:19:22.114127 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:22.132232 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:22.135264 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:22.135460 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:22.142208 18810 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38791
I20260928 21:19:22.142232 18877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38791 every 8 connection(s)
I20260928 21:19:22.143230 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:22.146075 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:22.150602 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18810
I20260928 21:19:22.150708 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:19:22.150974 5195 external_mini_cluster.cc:1428] Setting key eaf9aba7c6f4cfca192bd2b641cf22dd
I20260928 21:19:22.152068 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941: Bootstrap starting.
I20260928 21:19:22.154562 18878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:22.155324 18878 log.cc:824] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:22.157204 18878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941: No bootstrap required, opened a new log
Sep 28 21:19:22 dist-test-slave-rc9f krb5kdc[18797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:22.161230 18878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } }
I20260928 21:19:22.161492 18878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:22.161584 18878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3604480e534a56b0ac9876d99a6941, State: Initialized, Role: FOLLOWER
I20260928 21:19:22.162122 18878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } }
I20260928 21:19:22.162281 18878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:22.162348 18878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:22.162446 18878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:22.163530 18878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } }
I20260928 21:19:22.163906 18878 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3604480e534a56b0ac9876d99a6941; no voters:
I20260928 21:19:22.164266 18878 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:19:22.164383 18883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:22.164649 18883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [term 1 LEADER]: Becoming Leader. State: Replica: 7e3604480e534a56b0ac9876d99a6941, State: Running, Role: LEADER
I20260928 21:19:22.165024 18883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } }
I20260928 21:19:22.165699 18878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:22.165755 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:22.152375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.5.18.254:38791)
0928 21:19:22.152863 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:22.152879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:22.152917 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:19:22.153675 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:22.153699 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:22.154090 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:22.154432 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:22.155232 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:22.156253 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:22.157155 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:22.157747 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:22.157904 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:22.160596 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:22.160601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:22.160615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:22.160641 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:22.162584 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:22.162996 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:22.163002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:22.163008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:22.163151 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:22.163377 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:22.163380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:22.163382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:22.163926 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:22.164105 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:22.164376 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 21:19:22.164708 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260928 21:19:22.166723 18884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3604480e534a56b0ac9876d99a6941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } } }
I20260928 21:19:22.166723 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3604480e534a56b0ac9876d99a6941. Latest consensus state: current_term: 1 leader_uuid: "7e3604480e534a56b0ac9876d99a6941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3604480e534a56b0ac9876d99a6941" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38791 } } }
I20260928 21:19:22.166872 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:22.166872 18884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941 [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:22.167321 18892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:22.169049 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:19:22.170800 18892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:22.176321 18892 catalog_manager.cc:1347] Generated new cluster ID: 1da18b18e3374f9fbc8c6645ca19e72b
I20260928 21:19:22.176394 18892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:22.195309 18892 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:19:22.196350 18892 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:22.200582 18892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3604480e534a56b0ac9876d99a6941: Generated new TSK 0
I20260928 21:19:22.201294 18892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:19:22.224978 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18810
2026-09-28T21:19:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1335 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:19:22 dist-test-slave-rc9f krb5kdc[18905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:19:22 dist-test-slave-rc9f krb5kdc[18905](info): set up 2 sockets
Sep 28 21:19:22 dist-test-slave-rc9f krb5kdc[18905](info): commencing operation
krb5kdc: starting...
W20260928 21:19:23.177419 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:19:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:19:23.296685 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:23.406709 18918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:23.407042 18918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:23.407085 18918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:23.411147 18918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:23.411213 18918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:23.411231 18918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:23.411247 18918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:23.411260 18918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:23.416752 18918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:23.417870 18918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:23.418882 18918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:23.427713 18926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:23.427728 18923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:23.427801 18918 server_base.cc:1034] running on GCE node
W20260928 21:19:23.427712 18924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:23.428522 18918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:23.429688 18918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:23.430868 18918 hybrid_clock.cc:648] HybridClock initialized: now 1790630363430838 us; error 42 us; skew 500 ppm
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:23.433756 18918 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:23.434808 18918 webserver.cc:460] Webserver started at http://127.5.18.254:37211/ using document root <none> and password file <none>
I20260928 21:19:23.435380 18918 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:23.435446 18918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:23.435647 18918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:23.437885 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "eccae80ca5d7430996f6fde8db4dcebb"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "70ce3ecfd4b3562f29fddc88863736f0"
server_key_iv: "d4b95f2b9910e9b40fad8c163d76fc2e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.438477 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "eccae80ca5d7430996f6fde8db4dcebb"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "70ce3ecfd4b3562f29fddc88863736f0"
server_key_iv: "d4b95f2b9910e9b40fad8c163d76fc2e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.442420 18918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:19:23.445060 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.446456 18918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:19:23.446604 18918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "eccae80ca5d7430996f6fde8db4dcebb"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "70ce3ecfd4b3562f29fddc88863736f0"
server_key_iv: "d4b95f2b9910e9b40fad8c163d76fc2e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.446724 18918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:23.456772 18918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:23.460407 18918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:23.460608 18918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:23.467674 18918 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46147
I20260928 21:19:23.467941 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46147 every 8 connection(s)
I20260928 21:19:23.468707 18918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:23.471583 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:23.472899 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 18918
I20260928 21:19:23.473016 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:19:23.473322 5195 external_mini_cluster.cc:1428] Setting key 5ae414e5fe997c0503d7f6a2ac1d1cda
I20260928 21:19:23.477833 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap starting.
I20260928 21:19:23.480154 18986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:23.481006 18986 log.cc:824] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:23.483045 18986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: No bootstrap required, opened a new log
I20260928 21:19:23.486637 18986 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:23.486914 18986 raft_consensus.cc:374] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:23.486969 18986 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Initialized, Role: FOLLOWER
I20260928 21:19:23.487293 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:23.474802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.5.18.254:46147)
0928 21:19:23.475265 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:23.475284 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:23.475321 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:19:23.475968 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:23.475986 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:23.476337 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:23.476668 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:23.477463 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.478534 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.479399 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.479891 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.480040 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:23.482639 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:23.482644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:23.482660 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:23.482688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:23.484279 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.484703 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.484708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.484714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.484835 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.485089 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.485092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.485093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.485552 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:23.485786 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:23.486081 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:19:23.486390 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260928 21:19:23.487419 18986 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:23.487565 18986 raft_consensus.cc:388] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:23.487627 18986 raft_consensus.cc:482] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:23.487725 18986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:23.488827 18986 raft_consensus.cc:504] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:23.489194 18986 leader_election.cc:302] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eccae80ca5d7430996f6fde8db4dcebb; no voters:
I20260928 21:19:23.489524 18986 leader_election.cc:288] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:19:23.489696 18991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:23.489898 18991 raft_consensus.cc:686] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 LEADER]: Becoming Leader. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Running, Role: LEADER
I20260928 21:19:23.490231 18991 consensus_queue.cc:237] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:23.490824 18986 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:23.491783 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eccae80ca5d7430996f6fde8db4dcebb. Latest consensus state: current_term: 1 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:23.491905 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:23.492180 18992 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:23.492255 18992 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
W20260928 21:19:23.495988 19007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:19:23.496073 19007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:19:23.496169 19000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:23.496869 19000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:23.502189 19000 catalog_manager.cc:1347] Generated new cluster ID: 82cfb4982c424cdf89384200662c34be
I20260928 21:19:23.502282 19000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:23.513633 19000 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:19:23.514643 19000 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:23.519901 19000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Generated new TSK 0
I20260928 21:19:23.520622 19000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:19:23.560758 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:23.677966 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:23.678251 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:23.678296 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:23.682390 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:23.682466 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:23.682530 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:23.687978 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:23.689096 19012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:23.690143 19012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:19:23.699918 19012 server_base.cc:1034] running on GCE node
W20260928 21:19:23.699862 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:23.699862 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:23.699873 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:23.700587 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:23.701272 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:23.702433 19012 hybrid_clock.cc:648] HybridClock initialized: now 1790630363702394 us; error 52 us; skew 500 ppm
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:23.705164 19012 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:23.706151 19012 webserver.cc:460] Webserver started at http://127.5.18.193:35239/ using document root <none> and password file <none>
I20260928 21:19:23.706702 19012 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:23.706766 19012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:23.706954 19012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:23.708993 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "32d145f341f44bf09320076f8b43866e"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "747fd1e08e05d2a61f5065927025fd51"
server_key_iv: "95f4d9b4bf005d53d81b3b993c4d4d1e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.709524 19012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "32d145f341f44bf09320076f8b43866e"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "747fd1e08e05d2a61f5065927025fd51"
server_key_iv: "95f4d9b4bf005d53d81b3b993c4d4d1e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.713336 19012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:19:23.715857 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.717139 19012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:23.717257 19012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "32d145f341f44bf09320076f8b43866e"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "747fd1e08e05d2a61f5065927025fd51"
server_key_iv: "95f4d9b4bf005d53d81b3b993c4d4d1e"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.717368 19012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:23.751024 19012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:23.754238 19012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:23.754422 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:23.755020 19012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:23.755846 19012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:23.755892 19012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.755944 19012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:23.755975 19012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.764526 19012 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:40407
I20260928 21:19:23.764554 19140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:40407 every 8 connection(s)
I20260928 21:19:23.765457 19012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:23.767982 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19012
I20260928 21:19:23.768093 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:19:23.768376 5195 external_mini_cluster.cc:1428] Setting key 5e55fbcaa42ff88c357a4fb85a0fd77b
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:19:23.779340 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:23.767271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:58505 (local address 127.5.18.254:46147)
0928 21:19:23.767416 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:23.767421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:23.768338 (+ 917us) server_negotiation.cc:407] Connection header received
0928 21:19:23.769888 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:23.769891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:23.769943 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:23.770020 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:23.771155 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.771689 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.772827 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.773057 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.773145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:23.775870 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:23.775874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:23.775877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:23.775910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:23.777354 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.777885 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.777887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.777889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.777963 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.778302 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.778304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.778305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.778483 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:23.778553 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:23.779097 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 21:19:23.779200 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:19:23.780280 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:23.767704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:58505)
0928 21:19:23.768172 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:23.768225 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:23.769652 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:23.770164 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:23.770171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:23.770524 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:23.771016 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:23.771035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.771803 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.771807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:23.772711 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:23.772720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.773196 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.773200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:23.773382 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:23.774028 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:23.774044 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:23.775673 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:23.777461 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:23.777466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:23.777480 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:23.777810 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:23.778062 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:23.778065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:23.778067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:23.778229 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:23.778556 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:23.778562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:23.778819 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 21:19:23.779061 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 21:19:23.779351 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260928 21:19:23.781503 19141 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:23.781808 19141 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:23.782401 19141 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:23.783917 18950 ts_manager.cc:194] Registered new tserver with Master: 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407)
I20260928 21:19:23.785041 18950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:58505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:19:23.796620 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:23.904907 19146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:23.905194 19146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:23.905252 19146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:23.909301 19146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:23.909379 19146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:23.909471 19146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:23.914891 19146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:23.916002 19146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:23.917042 19146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:23.926523 19151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:23.926622 19146 server_base.cc:1034] running on GCE node
W20260928 21:19:23.926523 19152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:23.926571 19154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:23.927184 19146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:23.927824 19146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:23.928995 19146 hybrid_clock.cc:648] HybridClock initialized: now 1790630363928965 us; error 40 us; skew 500 ppm
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:23.931711 19146 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:23.932591 19146 webserver.cc:460] Webserver started at http://127.5.18.194:42361/ using document root <none> and password file <none>
I20260928 21:19:23.933139 19146 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:23.933203 19146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:23.933425 19146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:23.935504 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "341ff820a8494250ae716265fa887817"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "ae2f9da1968797cf35cb151b740c6a2b"
server_key_iv: "e377c3a521ef1f4ed6813cadc7057ac7"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.936038 19146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "341ff820a8494250ae716265fa887817"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "ae2f9da1968797cf35cb151b740c6a2b"
server_key_iv: "e377c3a521ef1f4ed6813cadc7057ac7"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.939891 19146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:19:23.942472 19161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.943799 19146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:23.943923 19146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "341ff820a8494250ae716265fa887817"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "ae2f9da1968797cf35cb151b740c6a2b"
server_key_iv: "e377c3a521ef1f4ed6813cadc7057ac7"
server_key_version: "encryptionkey@0"
I20260928 21:19:23.944046 19146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:23.958911 19146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:23.961859 19146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:23.962050 19146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:23.962664 19146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:23.963500 19146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:23.963547 19146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.963601 19146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:23.963631 19146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:23.971702 19146 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37991
I20260928 21:19:23.971742 19274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37991 every 8 connection(s)
I20260928 21:19:23.972601 19146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:19:23 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:23.982601 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19146
I20260928 21:19:23.982719 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:19:23.983050 5195 external_mini_cluster.cc:1428] Setting key 8405b78bbcadbde51fe13f315e264001
I20260928 21:19:23.985477 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:23.974385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:42527 (local address 127.5.18.254:46147)
0928 21:19:23.974549 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:23.974553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:23.975409 (+ 856us) server_negotiation.cc:407] Connection header received
0928 21:19:23.976595 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:23.976598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:23.976648 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:23.976719 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:23.977894 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.978415 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.979464 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.979669 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.979735 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:23.982030 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:23.982034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:23.982036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:23.982060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:23.983477 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.983997 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.983999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.984000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.984075 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:23.984423 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:23.984425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:23.984426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:23.984630 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:23.984724 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:23.985205 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 21:19:23.985323 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:19:23.986263 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:23.974820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:42527)
0928 21:19:23.975260 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:23.975303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:23.976406 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:23.976877 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:23.976884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:23.977259 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:23.977776 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:23.977790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.978502 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.978504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:23.979377 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:23.979385 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:23.979773 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:23.979775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:23.979913 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:23.980432 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:23.980448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:23.981876 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:23.983581 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:23.983587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:23.983603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:23.983891 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:23.984149 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:23.984151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:23.984153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:23.984335 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:23.984726 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:23.984731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:23.984976 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 21:19:23.985127 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:19:23.985378 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260928 21:19:23.987438 19275 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:23.987692 19275 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:23.988236 19275 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:23.989357 18950 ts_manager.cc:194] Registered new tserver with Master: 341ff820a8494250ae716265fa887817 (127.5.18.194:37991)
I20260928 21:19:23.990152 18950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:42527
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:19:24.012319 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:24.121976 19280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:24.122233 19280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:24.122290 19280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:24.126222 19280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:24.126279 19280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:24.126358 19280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:24.131505 19280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:24.132561 19280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:24.133673 19280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:24.143270 19288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.143313 19286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.143368 19285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.143589 19280 server_base.cc:1034] running on GCE node
I20260928 21:19:24.144032 19280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:24.144682 19280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:24.145823 19280 hybrid_clock.cc:648] HybridClock initialized: now 1790630364145793 us; error 42 us; skew 500 ppm
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:24.151799 19280 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:24.152789 19280 webserver.cc:460] Webserver started at http://127.5.18.195:45057/ using document root <none> and password file <none>
I20260928 21:19:24.153373 19280 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:24.153442 19280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:24.153682 19280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:24.155769 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
format_stamp: "Formatted at 2026-09-28 21:19:24 on dist-test-slave-rc9f"
server_key: "eead4fd1336b0c8ecb98b20a4ab4a224"
server_key_iv: "5ba793a7b88942cf88b90cbc58c165e4"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.156317 19280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
format_stamp: "Formatted at 2026-09-28 21:19:24 on dist-test-slave-rc9f"
server_key: "eead4fd1336b0c8ecb98b20a4ab4a224"
server_key_iv: "5ba793a7b88942cf88b90cbc58c165e4"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.160033 19280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:19:24.162487 19295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.163687 19280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:24.163792 19280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
format_stamp: "Formatted at 2026-09-28 21:19:24 on dist-test-slave-rc9f"
server_key: "eead4fd1336b0c8ecb98b20a4ab4a224"
server_key_iv: "5ba793a7b88942cf88b90cbc58c165e4"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.163889 19280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:24.173269 19280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:24.176162 19280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:24.176342 19280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:24.176889 19280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:24.177711 19280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:24.177758 19280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.177809 19280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:24.177837 19280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.186247 19280 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:35815
I20260928 21:19:24.186306 19408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:35815 every 8 connection(s)
I20260928 21:19:24.187126 19280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:19:24.188285 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19280
I20260928 21:19:24.188416 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:19:24.188709 5195 external_mini_cluster.cc:1428] Setting key c48765fb194126a4e1b29820609e880e
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:24.200441 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.188979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:32795 (local address 127.5.18.254:46147)
0928 21:19:24.189312 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.189318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.189991 (+ 673us) server_negotiation.cc:407] Connection header received
0928 21:19:24.191317 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.191319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.191373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.191456 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:24.192649 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.193259 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.194437 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.194656 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.194741 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.197101 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:24.197104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:24.197107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:24.197133 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:24.198497 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.198996 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.198999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.199000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.199073 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.199390 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.199393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.199395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.199575 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:24.199657 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.200150 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.200274 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 21:19:24.201375 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.189366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:32795)
0928 21:19:24.189842 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.189896 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.191097 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.191602 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.191609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.192007 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:24.192518 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.192533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.193368 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.193370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.194311 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.194321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.194784 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.194786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.194934 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.195484 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:24.195500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:24.196941 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:24.198607 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.198614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.198633 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.198905 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.199152 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.199156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.199158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.199298 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.199671 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:24.199678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:24.199928 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.200095 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.200444 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":132,"threads_started":1}
I20260928 21:19:24.202497 19409 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:24.202724 19409 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:24.203189 19409 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:24.204113 18950 ts_manager.cc:194] Registered new tserver with Master: 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815)
I20260928 21:19:24.204672 18950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:32795
I20260928 21:19:24.214745 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:19:24.223470 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.216643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.5.18.254:46147)
0928 21:19:24.216770 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.216775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.216903 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:19:24.217007 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.217009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.217050 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.217112 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:24.217805 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.218320 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.219165 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.219388 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.219464 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.220789 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:24.220792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:24.220794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:24.220831 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:24.222176 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.222557 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.222560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.222561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.222631 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.222810 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.222822 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.222823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.223024 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:24.223118 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.223263 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.223350 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:19:24.226284 18950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:19:24.229182 18950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:19:24.242799 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.237371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.0.0.1:56276)
0928 21:19:24.238260 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.238302 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.238412 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.239033 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.239039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.239061 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:24.239204 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.239211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.240285 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.240289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.241450 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.241460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.242276 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.242279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.242386 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.242423 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.242533 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.242652 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":94,"threads_started":1}
I20260928 21:19:24.242808 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.237493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.5.18.193:40407)
0928 21:19:24.238116 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.238132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.238587 (+ 455us) server_negotiation.cc:407] Connection header received
0928 21:19:24.238666 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.238671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.238890 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.239033 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:24.239312 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.240176 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.241587 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.242164 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.242263 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.242424 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.242563 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.242659 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":75,"threads_started":1}
I20260928 21:19:24.244228 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.237444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37991 (local address 127.0.0.1:51802)
0928 21:19:24.238258 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.238299 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.238410 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.239739 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.239742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.239759 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:24.239866 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.239873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.241318 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.241321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.242788 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.242795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.243872 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.243875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.243960 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.243981 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.244028 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.244096 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":124,"thread_start_us":133,"threads_started":1}
I20260928 21:19:24.244391 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.237687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:35815 (local address 127.0.0.1:44800)
0928 21:19:24.238995 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.239007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.239099 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.239617 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.239619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.239637 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:24.239736 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.239741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.241331 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.241333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.242777 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.242788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.243840 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.243844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.244084 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.244108 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.244175 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.244241 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":154,"thread_start_us":100,"threads_started":1}
I20260928 21:19:24.244822 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.238517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.5.18.195:35815)
0928 21:19:24.238860 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.238872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.239186 (+ 314us) server_negotiation.cc:407] Connection header received
0928 21:19:24.239264 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.239269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.239487 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.239629 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:24.239850 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.241200 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.242912 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.243734 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.243845 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.243982 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.244575 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.244674 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":132,"threads_started":1}
I20260928 21:19:24.244947 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.238955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.5.18.194:37991)
0928 21:19:24.239356 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.239367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.239386 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:19:24.239450 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.239455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.239606 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.240006 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:24.240149 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.241176 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.242912 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.243734 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.244411 (+ 677us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.244667 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.244715 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.244803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":114,"threads_started":1}
I20260928 21:19:24.246363 19075 tablet_service.cc:1467] Processing CreateTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 (DEFAULT_TABLE table=test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:24.247526 19075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e90586c50c8489f8ead8ddc4914b121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:24.247323 19343 tablet_service.cc:1467] Processing CreateTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 (DEFAULT_TABLE table=test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:24.247592 19209 tablet_service.cc:1467] Processing CreateTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 (DEFAULT_TABLE table=test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:24.248237 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e90586c50c8489f8ead8ddc4914b121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:24.248556 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e90586c50c8489f8ead8ddc4914b121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:24.253965 19426 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Bootstrap starting.
I20260928 21:19:24.255355 19427 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Bootstrap starting.
I20260928 21:19:24.256390 19426 tablet_bootstrap.cc:654] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:24.257063 19426 log.cc:824] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:24.257154 19427 tablet_bootstrap.cc:654] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:24.257952 19428 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Bootstrap starting.
I20260928 21:19:24.258263 19427 log.cc:824] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:24.259204 19426 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: No bootstrap required, opened a new log
I20260928 21:19:24.259406 19426 ts_tablet_manager.cc:1397] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:19:24.259984 19427 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: No bootstrap required, opened a new log
I20260928 21:19:24.260150 19427 ts_tablet_manager.cc:1397] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:19:24.260870 19428 tablet_bootstrap.cc:654] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:24.261688 19428 log.cc:824] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:24.263199 19427 raft_consensus.cc:348] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.263226 19426 raft_consensus.cc:348] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.263420 19427 raft_consensus.cc:374] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:24.263475 19427 raft_consensus.cc:729] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Initialized, Role: FOLLOWER
I20260928 21:19:24.263514 19426 raft_consensus.cc:374] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:24.263576 19426 raft_consensus.cc:729] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c14cf0b2abd4660b5335e03cd36d0f9, State: Initialized, Role: FOLLOWER
I20260928 21:19:24.263738 19428 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: No bootstrap required, opened a new log
I20260928 21:19:24.263963 19428 ts_tablet_manager.cc:1397] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:19:24.263927 19427 consensus_queue.cc:260] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.263993 19426 consensus_queue.cc:260] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.264504 19141 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
I20260928 21:19:24.264879 19427 ts_tablet_manager.cc:1428] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:19:24.264915 19426 ts_tablet_manager.cc:1428] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260928 21:19:24.265220 19409 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
W20260928 21:19:24.266577 19142 tablet.cc:2369] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:19:24.268646 19428 raft_consensus.cc:348] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.268998 19428 raft_consensus.cc:374] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:24.269065 19428 raft_consensus.cc:729] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341ff820a8494250ae716265fa887817, State: Initialized, Role: FOLLOWER
I20260928 21:19:24.269513 19428 consensus_queue.cc:260] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.270139 19275 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
I20260928 21:19:24.270428 19428 ts_tablet_manager.cc:1428] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:19:24.315220 19432 raft_consensus.cc:482] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:24.315438 19432 raft_consensus.cc:504] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.316592 19432 leader_election.cc:288] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341ff820a8494250ae716265fa887817 (127.5.18.194:37991), 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815)
I20260928 21:19:24.321444 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.317111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:54801 (local address 127.5.18.195:35815)
0928 21:19:24.317213 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.317216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.317380 (+ 164us) server_negotiation.cc:407] Connection header received
0928 21:19:24.317482 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.317485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.317534 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.317634 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:24.317856 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.318564 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320192 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.320790 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320909 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.320962 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.321233 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.321308 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:19:24.321511 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.316930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:41933 (local address 127.5.18.194:37991)
0928 21:19:24.317054 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.317059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.317150 (+ 91us) server_negotiation.cc:407] Connection header received
0928 21:19:24.317283 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.317286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.317334 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.317405 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:24.317714 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.318501 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320181 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.320791 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320909 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.320962 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.321324 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.321397 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:19:24.321552 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.316801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37991 (local address 127.5.18.193:41933)
0928 21:19:24.317060 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.317075 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.317183 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.317437 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.317440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.317455 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:24.317602 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.317608 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.318613 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.318618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.320085 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.320091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.320935 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.321067 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.321123 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.321203 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.321367 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":123,"thread_start_us":94,"threads_started":1}
I20260928 21:19:24.321554 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.316832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:35815 (local address 127.5.18.193:54801)
0928 21:19:24.317282 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.317294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.317394 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.317643 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.317646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.317663 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:24.317758 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.317763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.318654 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.318657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.320069 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.320081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.320935 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.320939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.321082 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.321123 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.321203 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.321367 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":171,"thread_start_us":120,"threads_started":1}
I20260928 21:19:24.322237 19363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e90586c50c8489f8ead8ddc4914b121" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" is_pre_election: true
I20260928 21:19:24.322239 19229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e90586c50c8489f8ead8ddc4914b121" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341ff820a8494250ae716265fa887817" is_pre_election: true
I20260928 21:19:24.322562 19363 raft_consensus.cc:2454] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 0.
I20260928 21:19:24.322562 19229 raft_consensus.cc:2454] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 0.
I20260928 21:19:24.323002 19030 leader_election.cc:302] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 341ff820a8494250ae716265fa887817; no voters:
I20260928 21:19:24.323298 19432 raft_consensus.cc:2781] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:19:24.323387 19432 raft_consensus.cc:482] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:24.323419 19432 raft_consensus.cc:3037] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:24.324429 19432 raft_consensus.cc:504] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.324764 19432 leader_election.cc:288] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 1 election: Requested vote from peers 341ff820a8494250ae716265fa887817 (127.5.18.194:37991), 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815)
I20260928 21:19:24.325101 19363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e90586c50c8489f8ead8ddc4914b121" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
I20260928 21:19:24.325212 19363 raft_consensus.cc:3037] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:24.325708 19229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e90586c50c8489f8ead8ddc4914b121" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341ff820a8494250ae716265fa887817"
I20260928 21:19:24.325821 19229 raft_consensus.cc:3037] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:24.326337 19363 raft_consensus.cc:2454] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 1.
I20260928 21:19:24.326613 19028 leader_election.cc:302] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 9c14cf0b2abd4660b5335e03cd36d0f9; no voters:
I20260928 21:19:24.326781 19432 raft_consensus.cc:2781] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:24.326937 19229 raft_consensus.cc:2454] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 1.
I20260928 21:19:24.326983 19432 raft_consensus.cc:686] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 LEADER]: Becoming Leader. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Running, Role: LEADER
I20260928 21:19:24.327342 19432 consensus_queue.cc:237] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.330453 18949 catalog_manager.cc:5495] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e reported cstate change: term changed from 0 to 1, leader changed from <none> to 32d145f341f44bf09320076f8b43866e (127.5.18.193). New cstate: current_term: 1 leader_uuid: "32d145f341f44bf09320076f8b43866e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260928 21:19:24.364586 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.361158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.5.18.193:40407)
0928 21:19:24.361295 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.361299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.361315 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:19:24.361397 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.361399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.361457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.361547 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:19:24.361811 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.362355 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.363185 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.363405 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.363490 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.363787 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:19:24.364219 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:19:24.364336 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.364390 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.364447 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:19:24.370909 19229 raft_consensus.cc:1261] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Refusing update from remote peer 32d145f341f44bf09320076f8b43866e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:24.371609 19437 consensus_queue.cc:1035] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [LEADER]: Connected to new peer: Peer: permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:24.371719 19363 raft_consensus.cc:1261] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Refusing update from remote peer 32d145f341f44bf09320076f8b43866e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:24.372248 19437 consensus_queue.cc:1035] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:24.380297 19442 mvcc.cc:205] Tried to move back new op lower bound from 7334421972455030784 to 7334421972287229952. Current Snapshot: MvccSnapshot[applied={T|T < 7334421972455030784}]
I20260928 21:19:24.387761 18949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:24.387972 18949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:24.390553 18949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Sending DeleteTablet for 3 replicas of tablet 2e90586c50c8489f8ead8ddc4914b121
I20260928 21:19:24.391357 19343 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:24 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:44800
I20260928 21:19:24.391342 19209 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:24 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:51802
I20260928 21:19:24.391341 19075 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:24 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:56276
I20260928 21:19:24.391868 19450 tablet_replica.cc:330] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: stopping tablet replica
I20260928 21:19:24.392186 19450 raft_consensus.cc:2229] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:24.392354 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19012
I20260928 21:19:24.392555 19450 raft_consensus.cc:2258] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:24.393205 19451 tablet_replica.cc:330] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: stopping tablet replica
I20260928 21:19:24.393514 19451 raft_consensus.cc:2229] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:24.393855 19451 raft_consensus.cc:2258] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:24.394644 19450 ts_tablet_manager.cc:1905] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:24.395205 19451 ts_tablet_manager.cc:1905] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:24.397344 19451 ts_tablet_manager.cc:1918] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:24.397401 19450 ts_tablet_manager.cc:1918] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:24.397435 19451 log.cc:1196] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/2e90586c50c8489f8ead8ddc4914b121
I20260928 21:19:24.397524 19450 log.cc:1196] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/2e90586c50c8489f8ead8ddc4914b121
I20260928 21:19:24.397758 19451 ts_tablet_manager.cc:1939] T 2e90586c50c8489f8ead8ddc4914b121 P 341ff820a8494250ae716265fa887817: Deleting consensus metadata
I20260928 21:19:24.397866 19450 ts_tablet_manager.cc:1939] T 2e90586c50c8489f8ead8ddc4914b121 P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting consensus metadata
I20260928 21:19:24.398981 18936 catalog_manager.cc:4841] TS 341ff820a8494250ae716265fa887817 (127.5.18.194:37991): tablet 2e90586c50c8489f8ead8ddc4914b121 (table test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]) successfully deleted
I20260928 21:19:24.399065 18934 catalog_manager.cc:4841] TS 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815): tablet 2e90586c50c8489f8ead8ddc4914b121 (table test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]) successfully deleted
I20260928 21:19:24.400008 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19146
W20260928 21:19:24.400007 18935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:40407 (error 108)
I20260928 21:19:24.402401 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.402133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.0.0.1:56300)
0928 21:19:24.402261 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.402328 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:19:24.402546 18935 catalog_manager.cc:4584] TS 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e90586c50c8489f8ead8ddc4914b121: Network error: Client connection negotiation failed: client connection to 127.5.18.193:40407: connect: Connection refused (error 111)
I20260928 21:19:24.404947 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19280
I20260928 21:19:24.409858 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 18918
I20260928 21:19:24.415320 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--webserver_interface=127.5.18.254
--webserver_port=37211
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:24.526773 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:24.527048 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:24.527091 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:24.531145 19454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:24.531216 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:24.531234 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:24.531250 19454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:24.531263 19454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:24.536643 19454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:24.537803 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:24.538856 19454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:24.551843 19460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.551872 19463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.551961 19454 server_base.cc:1034] running on GCE node
W20260928 21:19:24.551843 19461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.552789 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:24.554041 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:24.555212 19454 hybrid_clock.cc:648] HybridClock initialized: now 1790630364555173 us; error 54 us; skew 500 ppm
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:24.558211 19454 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:24.559198 19454 webserver.cc:460] Webserver started at http://127.5.18.254:37211/ using document root <none> and password file <none>
I20260928 21:19:24.559763 19454 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:24.559832 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:24.563836 19454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:19:24.565995 19470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.567272 19454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:24.567392 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "eccae80ca5d7430996f6fde8db4dcebb"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "70ce3ecfd4b3562f29fddc88863736f0"
server_key_iv: "d4b95f2b9910e9b40fad8c163d76fc2e"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.568029 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:24.583137 19454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:24.586727 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:24.586930 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:24.594533 19454 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46147
I20260928 21:19:24.594825 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46147 every 8 connection(s)
I20260928 21:19:24.595764 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:24.600746 19523 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:19:24.602030 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19454
I20260928 21:19:24.602452 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:40407
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=35239
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:24.602854 19523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap starting.
I20260928 21:19:24.620744 19523 log.cc:824] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:24.630501 19523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:24.630913 19523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap complete.
I20260928 21:19:24.635040 19523 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:24.635492 19523 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Initialized, Role: FOLLOWER
I20260928 21:19:24.636000 19523 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:24.636147 19523 raft_consensus.cc:388] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:24.636230 19523 raft_consensus.cc:482] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:24.636351 19523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:19:24.638530 19523 raft_consensus.cc:504] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:24.638890 19523 leader_election.cc:302] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eccae80ca5d7430996f6fde8db4dcebb; no voters:
I20260928 21:19:24.639228 19523 leader_election.cc:288] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:19:24.639348 19527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:19:24.639568 19527 raft_consensus.cc:686] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 LEADER]: Becoming Leader. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Running, Role: LEADER
I20260928 21:19:24.639976 19527 consensus_queue.cc:237] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:24.640648 19523 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:24.641399 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:24.641534 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:24.642032 19529 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eccae80ca5d7430996f6fde8db4dcebb. Latest consensus state: current_term: 2 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:24.642194 19529 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:24.642516 19536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:24.647410 19536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:24.648190 19536 catalog_manager.cc:1259] Loaded cluster ID: 82cfb4982c424cdf89384200662c34be
I20260928 21:19:24.648262 19536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:24.649782 19536 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 21:19:24.654153 19536 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 21:19:24.660315 19536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:24.661232 19536 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Loaded TSK: 0
I20260928 21:19:24.662576 19536 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:19:24.720286 19525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:24.720614 19525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:24.720683 19525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:24.724828 19525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:24.724911 19525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:24.724996 19525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:24.730850 19525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:40407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=35239
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:24.731963 19525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:24.733018 19525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:24.743780 19550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.743780 19551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.743813 19553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.744591 19525 server_base.cc:1034] running on GCE node
I20260928 21:19:24.745086 19525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:24.745927 19525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:24.747110 19525 hybrid_clock.cc:648] HybridClock initialized: now 1790630364747068 us; error 63 us; skew 500 ppm
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:24.750128 19525 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:24.751052 19525 webserver.cc:460] Webserver started at http://127.5.18.193:35239/ using document root <none> and password file <none>
I20260928 21:19:24.751605 19525 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:24.751685 19525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:24.755576 19525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:19:24.757754 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.759097 19525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:24.759204 19525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "32d145f341f44bf09320076f8b43866e"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "747fd1e08e05d2a61f5065927025fd51"
server_key_iv: "95f4d9b4bf005d53d81b3b993c4d4d1e"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.759763 19525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:24.779030 19525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:24.782994 19525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:24.783219 19525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:24.783936 19525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:24.785218 19567 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:19:24.789898 19525 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:19:24.789981 19525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 21:19:24.790040 19525 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:19:24.791904 19525 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:19:24.791980 19525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:19:24.792341 19567 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Bootstrap starting.
I20260928 21:19:24.801330 19525 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:40407
I20260928 21:19:24.801755 19674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:40407 every 8 connection(s)
I20260928 21:19:24.802549 19525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:24.809288 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19525
I20260928 21:19:24.809998 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:37991
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=42361
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:24.814481 19567 log.cc:824] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:24.824576 19567 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:24.825099 19567 tablet_bootstrap.cc:492] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Bootstrap complete.
I20260928 21:19:24.825344 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.805223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:58257)
0928 21:19:24.805889 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.805950 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.806895 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.808607 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.808615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.809163 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:24.810123 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.810150 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.812125 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.812128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.813238 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.813254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.814586 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.814591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.814879 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.815744 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:24.815773 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:24.818217 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:24.821375 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.821383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.821399 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.821856 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.822386 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.822390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.822392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.822570 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.823348 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:24.823355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:24.823650 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.823854 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.824225 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":161,"threads_started":1}
I20260928 21:19:24.825340 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.805211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:58257 (local address 127.5.18.254:46147)
0928 21:19:24.805722 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.805748 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.806103 (+ 355us) server_negotiation.cc:407] Connection header received
0928 21:19:24.807488 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.807507 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.808025 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.808523 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:24.810387 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.811903 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.813449 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.814388 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.814611 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.818600 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:24.818610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:24.818636 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:24.818686 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:24.821195 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.822027 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.822036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.822044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.822268 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.822683 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.822686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.822688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.823228 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:24.823365 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.824011 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.824343 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":155,"threads_started":1}
I20260928 21:19:24.826567 19567 ts_tablet_manager.cc:1397] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260928 21:19:24.827943 19675 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:24.828207 19675 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:24.828845 19675 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:24.829432 19567 raft_consensus.cc:348] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.830000 19567 raft_consensus.cc:729] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Initialized, Role: FOLLOWER
I20260928 21:19:24.830538 19567 consensus_queue.cc:260] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } }
I20260928 21:19:24.831580 19567 ts_tablet_manager.cc:1428] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:19:24.832108 19487 ts_manager.cc:194] Registered new tserver with Master: 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407)
I20260928 21:19:24.836709 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:58257
I20260928 21:19:24.839762 19675 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:19:24.848019 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.836162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.5.18.193:40407)
0928 21:19:24.836606 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:24.836618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:24.837059 (+ 441us) server_negotiation.cc:407] Connection header received
0928 21:19:24.837265 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:24.837292 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:24.837482 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:24.837668 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:24.838552 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.839590 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.840817 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.841331 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.841438 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.844248 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:24.844259 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:24.844272 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:24.844305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:24.845977 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.846569 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.846573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.846575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.846661 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:24.846927 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:24.846929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:24.846930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:24.847193 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:24.847325 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:24.847704 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 21:19:24.847878 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":93,"threads_started":1}
I20260928 21:19:24.848030 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:24.836073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.0.0.1:56310)
0928 21:19:24.836891 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:24.836945 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:24.837111 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:24.837792 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:24.837800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:24.838260 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:24.838401 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.838410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.839723 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.839728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.840664 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:24.840678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:24.841604 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:24.841608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:24.841725 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:24.842312 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:24.842327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:24.844071 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:24.846096 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.846109 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.846116 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.846446 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.846743 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:24.846746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:24.846748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:24.846848 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:24.847293 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:24.847300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:24.847448 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 21:19:24.847610 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 21:19:24.847844 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":131,"threads_started":1}
I20260928 21:19:24.849502 19609 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e90586c50c8489f8ead8ddc4914b121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:24 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:56310
I20260928 21:19:24.850454 19684 tablet_replica.cc:330] stopping tablet replica
I20260928 21:19:24.850790 19684 raft_consensus.cc:2229] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:24.851048 19684 raft_consensus.cc:2258] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:24.852131 19684 ts_tablet_manager.cc:1905] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:24.856088 19684 ts_tablet_manager.cc:1918] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:24.856199 19684 log.cc:1196] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/2e90586c50c8489f8ead8ddc4914b121
I20260928 21:19:24.856429 19684 ts_tablet_manager.cc:1939] T 2e90586c50c8489f8ead8ddc4914b121 P 32d145f341f44bf09320076f8b43866e: Deleting consensus metadata
I20260928 21:19:24.858440 19472 catalog_manager.cc:4841] TS 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407): tablet 2e90586c50c8489f8ead8ddc4914b121 (table test-table [id=d6569f2f150f44ecbbfe97ed22f3f973]) successfully deleted
W20260928 21:19:24.934849 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:24.935148 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:24.935192 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:24.939229 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:24.939292 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:24.939354 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:24.945118 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=42361
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:24.946520 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:24.947871 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:24.958283 19689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.958390 19679 server_base.cc:1034] running on GCE node
W20260928 21:19:24.958247 19692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:24.958271 19690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:24.958992 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:24.959659 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:24.960819 19679 hybrid_clock.cc:648] HybridClock initialized: now 1790630364960789 us; error 46 us; skew 500 ppm
Sep 28 21:19:24 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:24.963697 19679 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:24.964649 19679 webserver.cc:460] Webserver started at http://127.5.18.194:42361/ using document root <none> and password file <none>
I20260928 21:19:24.965211 19679 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:24.965293 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:24.969403 19679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260928 21:19:24.971789 19699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.973140 19679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:19:24.973266 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "341ff820a8494250ae716265fa887817"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "ae2f9da1968797cf35cb151b740c6a2b"
server_key_iv: "e377c3a521ef1f4ed6813cadc7057ac7"
server_key_version: "encryptionkey@0"
I20260928 21:19:24.973912 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:24.993422 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:24.996670 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:24.996857 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:24.997462 19679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:24.998315 19679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:24.998363 19679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:24.998414 19679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:24.998443 19679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.007164 19679 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37991
I20260928 21:19:25.007189 19812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37991 every 8 connection(s)
I20260928 21:19:25.008272 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:19:25.017908 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19679
Sep 28 21:19:25 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:25.018368 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:35815
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=45057
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:25.022668 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.010089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:47689 (local address 127.5.18.254:46147)
0928 21:19:25.010214 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.010218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.011128 (+ 910us) server_negotiation.cc:407] Connection header received
0928 21:19:25.012445 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.012448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.012525 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.012598 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:25.013949 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.014503 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.015748 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.015984 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.016099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.018775 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:25.018779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:25.018781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:25.018811 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:25.020395 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.020956 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.020959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.020960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.021044 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.021369 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.021372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.021373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.021603 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:25.021689 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.022297 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.022455 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:19:25.023747 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.010500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:47689)
0928 21:19:25.010975 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.011021 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.012207 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.012811 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.012820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.013217 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:25.013789 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.013805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.014637 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.014640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.015624 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.015634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.016140 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.016144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.016328 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.016912 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:25.016929 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:25.018574 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:25.020553 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:25.020559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:25.020574 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:25.020844 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:25.021131 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:25.021134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:25.021136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:25.021280 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:25.021708 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:25.021714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:25.022030 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.022252 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.022700 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260928 21:19:25.025177 19813 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:25.025478 19813 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:25.026063 19813 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:25.027305 19487 ts_manager.cc:194] Registered new tserver with Master: 341ff820a8494250ae716265fa887817 (127.5.18.194:37991)
I20260928 21:19:25.028138 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:47689
W20260928 21:19:25.132597 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:25.132908 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:25.132956 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:25.137732 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:25.137818 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:25.137882 19816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:25.143700 19816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:35815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=45057
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:25.144780 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:25.145848 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:25.155812 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:25.155821 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:25.155887 19816 server_base.cc:1034] running on GCE node
W20260928 21:19:25.155830 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:25.156495 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:25.157207 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:25.158370 19816 hybrid_clock.cc:648] HybridClock initialized: now 1790630365158344 us; error 42 us; skew 500 ppm
Sep 28 21:19:25 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:25.161079 19816 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:25.162034 19816 webserver.cc:460] Webserver started at http://127.5.18.195:45057/ using document root <none> and password file <none>
I20260928 21:19:25.162555 19816 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:25.162623 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:25.166618 19816 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:19:25.168699 19831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.170064 19816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:25.170172 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
format_stamp: "Formatted at 2026-09-28 21:19:24 on dist-test-slave-rc9f"
server_key: "eead4fd1336b0c8ecb98b20a4ab4a224"
server_key_iv: "5ba793a7b88942cf88b90cbc58c165e4"
server_key_version: "encryptionkey@0"
I20260928 21:19:25.170740 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:25.180976 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:25.184288 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:25.184469 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:25.185091 19816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:25.185925 19816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:25.185978 19816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.186018 19816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:25.186038 19816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.194480 19816 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:35815
I20260928 21:19:25.194515 19944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:35815 every 8 connection(s)
I20260928 21:19:25.195533 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:19:25 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:25.205070 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19816
I20260928 21:19:25.208999 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.197407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:49183 (local address 127.5.18.254:46147)
0928 21:19:25.197553 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.197557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.198451 (+ 894us) server_negotiation.cc:407] Connection header received
0928 21:19:25.199700 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.199703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.199754 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.199829 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:25.201041 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.201646 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.202742 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.202966 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.203043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.205481 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:25.205485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:25.205488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:25.205518 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:25.207026 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.207539 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.207542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.207543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.207618 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.207930 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.207933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.207935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.208106 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:25.208188 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.208705 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.208827 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:19:25.209998 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.197848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:49183)
0928 21:19:25.198291 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.198335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.199476 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.199994 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.200001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.200365 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:25.200890 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.200905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.201741 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.201744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.202643 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.202651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.203084 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.203087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.203221 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.203760 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:25.203776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:25.205296 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:25.207135 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:25.207142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:25.207155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:25.207442 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:25.207702 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:25.207705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:25.207706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:25.207845 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:25.208200 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:25.208208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:25.208479 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.208648 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.208992 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260928 21:19:25.211212 19945 heartbeater.cc:347] Connected to a master server at 127.5.18.254:46147
I20260928 21:19:25.211491 19945 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:25.212072 19945 heartbeater.cc:511] Master 127.5.18.254:46147 requested a full tablet report, sending...
I20260928 21:19:25.213263 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.205762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.5.18.254:46147)
0928 21:19:25.206037 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.206040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.206309 (+ 269us) server_negotiation.cc:407] Connection header received
0928 21:19:25.206463 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.206466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.206514 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.206584 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:25.207263 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.207793 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.208622 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.208857 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.208939 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.210257 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:25.210261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:25.210264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:25.210288 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:25.211733 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.212128 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.212132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.212133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.212214 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:25.212440 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:25.212443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:25.212444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:25.212656 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:25.212801 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.212998 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.213114 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260928 21:19:25.213282 19487 ts_manager.cc:194] Registered new tserver with Master: 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815)
I20260928 21:19:25.213999 19487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:49183
I20260928 21:19:25.215131 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:19:25.219347 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.215732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.5.18.254:46147)
0928 21:19:25.215854 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.215858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.216069 (+ 211us) server_negotiation.cc:407] Connection header received
0928 21:19:25.216193 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.216195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.216298 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.216421 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:19:25.216596 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.217101 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.218025 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.218260 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.218339 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.218623 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:19:25.219027 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:19:25.219115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.219144 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.219203 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:19:25.220008 19487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:19:25.223259 19487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:19:25.230523 19609 tablet_service.cc:1467] Processing CreateTablet for tablet 058408019f854f3c8d21a8ff9f8046da (DEFAULT_TABLE table=test-table [id=2ae82d2b20f54828a0113fad294aed95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:25.231218 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058408019f854f3c8d21a8ff9f8046da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:25.234282 19567 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Bootstrap starting.
I20260928 21:19:25.234769 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.229785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37991 (local address 127.0.0.1:51808)
0928 21:19:25.230170 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.230183 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.230262 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.231027 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.231030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.231049 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:25.231129 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.231135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.232368 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.232371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.233406 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.233413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.234297 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.234300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.234407 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.234463 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.234530 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.234610 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260928 21:19:25.235020 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.229485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:35815 (local address 127.0.0.1:44816)
0928 21:19:25.229670 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.229684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.229765 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.230698 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.230701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.230719 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:25.230827 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.230834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.231902 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.231906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.233137 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.233148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.234298 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.234302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.234720 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.234737 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.234805 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.234889 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":303}
I20260928 21:19:25.235255 19567 tablet_bootstrap.cc:654] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:25.235488 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.230247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.5.18.194:37991)
0928 21:19:25.230590 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.230601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.230622 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:19:25.230696 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.230702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.230894 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.231051 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:25.231212 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.232253 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.233521 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.234157 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.235043 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.235190 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.235240 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.235340 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260928 21:19:25.235692 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.229666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.5.18.195:35815)
0928 21:19:25.230181 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.230195 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.230218 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:19:25.230301 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.230306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.230555 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.230706 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:25.230935 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.231779 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.233254 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.234163 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.234479 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.234635 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.235055 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.235560 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":175,"threads_started":1}
I20260928 21:19:25.236785 19567 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: No bootstrap required, opened a new log
I20260928 21:19:25.236884 19567 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:19:25.237340 19567 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.237478 19567 raft_consensus.cc:374] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:25.237524 19567 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Initialized, Role: FOLLOWER
I20260928 21:19:25.237672 19567 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.237941 19567 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:19:25.238502 19879 tablet_service.cc:1467] Processing CreateTablet for tablet 058408019f854f3c8d21a8ff9f8046da (DEFAULT_TABLE table=test-table [id=2ae82d2b20f54828a0113fad294aed95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:25.238421 19747 tablet_service.cc:1467] Processing CreateTablet for tablet 058408019f854f3c8d21a8ff9f8046da (DEFAULT_TABLE table=test-table [id=2ae82d2b20f54828a0113fad294aed95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:25.239490 19747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058408019f854f3c8d21a8ff9f8046da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:25.239533 19879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058408019f854f3c8d21a8ff9f8046da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:25.246802 19959 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Bootstrap starting.
I20260928 21:19:25.246651 19958 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Bootstrap starting.
I20260928 21:19:25.248744 19959 tablet_bootstrap.cc:654] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:25.249068 19958 tablet_bootstrap.cc:654] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:25.249768 19959 log.cc:824] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:25.249774 19958 log.cc:824] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:25.251662 19959 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: No bootstrap required, opened a new log
I20260928 21:19:25.251662 19958 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: No bootstrap required, opened a new log
I20260928 21:19:25.251843 19959 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 21:19:25.251844 19958 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:19:25.254725 19959 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.254725 19958 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.254956 19958 raft_consensus.cc:374] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:25.254971 19959 raft_consensus.cc:374] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:25.254997 19958 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341ff820a8494250ae716265fa887817, State: Initialized, Role: FOLLOWER
I20260928 21:19:25.255026 19959 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c14cf0b2abd4660b5335e03cd36d0f9, State: Initialized, Role: FOLLOWER
I20260928 21:19:25.255419 19959 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.255419 19958 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.256078 19813 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
I20260928 21:19:25.256240 19945 heartbeater.cc:503] Master 127.5.18.254:46147 was elected leader, sending a full tablet report...
I20260928 21:19:25.256368 19958 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 21:19:25.257387 19959 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
W20260928 21:19:25.259212 19814 tablet.cc:2369] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:19:25.303912 19676 tablet.cc:2369] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:19:25.446488 19946 tablet.cc:2369] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:19:25.470108 19962 raft_consensus.cc:482] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:25.470301 19962 raft_consensus.cc:504] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.471465 19962 leader_election.cc:288] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815), 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407)
I20260928 21:19:25.475359 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.471734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.5.18.194:34091)
0928 21:19:25.471926 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.471941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.472037 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.472380 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.472383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.472399 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:25.472521 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.472527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.473290 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.473293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.474273 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.474282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.474929 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.475033 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.475054 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.475113 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.475187 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":33}
I20260928 21:19:25.475384 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.471880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:58841 (local address 127.5.18.195:35815)
0928 21:19:25.472037 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.472042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.472221 (+ 179us) server_negotiation.cc:407] Connection header received
0928 21:19:25.472343 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.472346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.472387 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.472448 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:25.472709 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.473190 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474461 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.474867 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.475019 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.475191 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.475265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:19:25.475397 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.471751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:35815 (local address 127.5.18.194:58841)
0928 21:19:25.472115 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.472127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.472244 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:25.472473 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:25.472475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:25.472491 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:25.472597 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.472602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.473294 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.473298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.474356 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:25.474365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.474971 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:25.475080 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.475101 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:19:25.475165 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:19:25.475233 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":17,"thread_start_us":108,"threads_started":1}
I20260928 21:19:25.475404 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.471855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:34091 (local address 127.5.18.193:40407)
0928 21:19:25.472134 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.472138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.472150 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:19:25.472195 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.472197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.472262 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.472359 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:25.472631 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.473193 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474385 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.474819 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.474929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.475125 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.475160 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.475290 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
I20260928 21:19:25.476008 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "341ff820a8494250ae716265fa887817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" is_pre_election: true
I20260928 21:19:25.476065 19629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "341ff820a8494250ae716265fa887817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d145f341f44bf09320076f8b43866e" is_pre_election: true
I20260928 21:19:25.476325 19629 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341ff820a8494250ae716265fa887817 in term 0.
I20260928 21:19:25.476364 19899 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341ff820a8494250ae716265fa887817 in term 0.
I20260928 21:19:25.476797 19701 leader_election.cc:302] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 341ff820a8494250ae716265fa887817; no voters:
I20260928 21:19:25.477048 19962 raft_consensus.cc:2781] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:19:25.477126 19962 raft_consensus.cc:482] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:25.477174 19962 raft_consensus.cc:3037] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:25.478245 19962 raft_consensus.cc:504] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.478581 19962 leader_election.cc:288] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [CANDIDATE]: Term 1 election: Requested vote from peers 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815), 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407)
I20260928 21:19:25.478953 19629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "341ff820a8494250ae716265fa887817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d145f341f44bf09320076f8b43866e"
I20260928 21:19:25.479094 19629 raft_consensus.cc:3037] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:25.479161 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "341ff820a8494250ae716265fa887817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
I20260928 21:19:25.479291 19899 raft_consensus.cc:3037] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:25.480124 19629 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341ff820a8494250ae716265fa887817 in term 1.
I20260928 21:19:25.480427 19899 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341ff820a8494250ae716265fa887817 in term 1.
I20260928 21:19:25.480438 19701 leader_election.cc:302] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 341ff820a8494250ae716265fa887817; no voters:
I20260928 21:19:25.480597 19962 raft_consensus.cc:2781] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:25.480844 19962 raft_consensus.cc:686] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 LEADER]: Becoming Leader. State: Replica: 341ff820a8494250ae716265fa887817, State: Running, Role: LEADER
I20260928 21:19:25.481170 19962 consensus_queue.cc:237] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:25.484565 19487 catalog_manager.cc:5495] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341ff820a8494250ae716265fa887817 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "341ff820a8494250ae716265fa887817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20260928 21:19:25.533958 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.530517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.5.18.194:37991)
0928 21:19:25.530652 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:25.530657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:25.530671 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:19:25.530777 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:25.530780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:25.530841 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:25.530945 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:19:25.531165 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.531673 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.532490 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:25.532702 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:25.532786 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:25.533095 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:19:25.533533 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:19:25.533684 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:25.533715 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 21:19:25.533778 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:19:25.540072 19899 raft_consensus.cc:1261] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Refusing update from remote peer 341ff820a8494250ae716265fa887817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:25.540068 19629 raft_consensus.cc:1261] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Refusing update from remote peer 341ff820a8494250ae716265fa887817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:25.540766 19962 consensus_queue.cc:1035] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:25.540967 19966 consensus_queue.cc:1035] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:25.547359 19969 mvcc.cc:205] Tried to move back new op lower bound from 7334421977244262400 to 7334421977013985280. Current Snapshot: MvccSnapshot[applied={T|T < 7334421977244262400}]
I20260928 21:19:25.548480 19970 mvcc.cc:205] Tried to move back new op lower bound from 7334421977244262400 to 7334421977013985280. Current Snapshot: MvccSnapshot[applied={T|T < 7334421977244262400}]
I20260928 21:19:25.556581 19485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:25.556747 19485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:25.559070 19485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Sending DeleteTablet for 3 replicas of tablet 058408019f854f3c8d21a8ff9f8046da
I20260928 21:19:25.559880 19609 tablet_service.cc:1514] Processing DeleteTablet for tablet 058408019f854f3c8d21a8ff9f8046da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:25 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:56310
I20260928 21:19:25.559824 19879 tablet_service.cc:1514] Processing DeleteTablet for tablet 058408019f854f3c8d21a8ff9f8046da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:25 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:44816
I20260928 21:19:25.559902 19747 tablet_service.cc:1514] Processing DeleteTablet for tablet 058408019f854f3c8d21a8ff9f8046da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:25 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:51808
I20260928 21:19:25.560485 19979 tablet_replica.cc:330] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: stopping tablet replica
I20260928 21:19:25.560480 19980 tablet_replica.cc:330] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: stopping tablet replica
I20260928 21:19:25.560528 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19525
I20260928 21:19:25.560628 19979 raft_consensus.cc:2229] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:25.560778 19979 raft_consensus.cc:2258] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:25.561030 19980 raft_consensus.cc:2229] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:25.561344 19980 raft_consensus.cc:2258] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:25.562790 19980 ts_tablet_manager.cc:1905] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:25.565045 19980 ts_tablet_manager.cc:1918] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:25.565161 19980 log.cc:1196] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/058408019f854f3c8d21a8ff9f8046da
I20260928 21:19:25.565455 19981 tablet_replica.cc:330] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: stopping tablet replica
I20260928 21:19:25.565508 19980 ts_tablet_manager.cc:1939] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Deleting consensus metadata
I20260928 21:19:25.565892 19981 raft_consensus.cc:2229] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:19:25.566387 19981 raft_consensus.cc:2258] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:25.566912 19471 catalog_manager.cc:4841] TS 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815): tablet 058408019f854f3c8d21a8ff9f8046da (table test-table [id=2ae82d2b20f54828a0113fad294aed95]) successfully deleted
W20260928 21:19:25.567668 19472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:40407 (error 108)
I20260928 21:19:25.568109 19981 ts_tablet_manager.cc:1905] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:19:25.568979 19682 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:19:25.568645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.0.0.1:56320)
0928 21:19:25.568794 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.568866 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:40407: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:19:25.569145 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19679
W20260928 21:19:25.571595 19472 catalog_manager.cc:4584] TS 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 058408019f854f3c8d21a8ff9f8046da: Network error: Client connection negotiation failed: client connection to 127.5.18.193:40407: connect: Connection reset by peer (error 104)
I20260928 21:19:25.576326 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19816
I20260928 21:19:25.576555 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:25.576277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37991 (local address 127.0.0.1:51826)
0928 21:19:25.576408 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.576460 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:37991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:19:25.581372 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 19454
I20260928 21:19:25.587344 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--webserver_interface=127.5.18.254
--webserver_port=37211
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:25.699102 19983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:25.699404 19983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:25.699445 19983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:25.703524 19983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:25.703589 19983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:25.703608 19983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:25.703626 19983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:25.703655 19983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:25.709036 19983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=37211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:25.710228 19983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:25.711265 19983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:25.719676 19989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:25.719732 19983 server_base.cc:1034] running on GCE node
W20260928 21:19:25.719673 19991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:25.719658 19988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:25.720526 19983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:25.721678 19983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:25.722837 19983 hybrid_clock.cc:648] HybridClock initialized: now 1790630365722818 us; error 34 us; skew 500 ppm
Sep 28 21:19:25 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:25.725489 19983 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:25.726383 19983 webserver.cc:460] Webserver started at http://127.5.18.254:37211/ using document root <none> and password file <none>
I20260928 21:19:25.726934 19983 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:25.727001 19983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:25.731071 19983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:19:25.733296 19998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.734653 19983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:25.734782 19983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "eccae80ca5d7430996f6fde8db4dcebb"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "70ce3ecfd4b3562f29fddc88863736f0"
server_key_iv: "d4b95f2b9910e9b40fad8c163d76fc2e"
server_key_version: "encryptionkey@0"
I20260928 21:19:25.735389 19983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:25.748860 19983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:25.752002 19983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:25.752192 19983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:25.759176 19983 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46147
I20260928 21:19:25.759459 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46147 every 8 connection(s)
I20260928 21:19:25.760308 19983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:25.763301 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 19983
I20260928 21:19:25.763691 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:40407
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=35239
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:25.764829 20051 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:19:25.767087 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap starting.
I20260928 21:19:25.790568 20051 log.cc:824] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:25.804581 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:25.805037 20051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb: Bootstrap complete.
I20260928 21:19:25.810498 20051 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:25.811020 20051 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Initialized, Role: FOLLOWER
I20260928 21:19:25.811620 20051 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:25.811774 20051 raft_consensus.cc:388] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:25.811847 20051 raft_consensus.cc:482] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:25.811941 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:19:25.814510 20051 raft_consensus.cc:504] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:25.814879 20051 leader_election.cc:302] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eccae80ca5d7430996f6fde8db4dcebb; no voters:
I20260928 21:19:25.815225 20051 leader_election.cc:288] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:19:25.815372 20055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:19:25.815603 20055 raft_consensus.cc:686] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [term 3 LEADER]: Becoming Leader. State: Replica: eccae80ca5d7430996f6fde8db4dcebb, State: Running, Role: LEADER
I20260928 21:19:25.815914 20055 consensus_queue.cc:237] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } }
I20260928 21:19:25.816599 20051 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:25.817339 20056 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:25.817488 20056 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:25.817337 20057 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eccae80ca5d7430996f6fde8db4dcebb. Latest consensus state: current_term: 3 leader_uuid: "eccae80ca5d7430996f6fde8db4dcebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccae80ca5d7430996f6fde8db4dcebb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46147 } } }
I20260928 21:19:25.817621 20057 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccae80ca5d7430996f6fde8db4dcebb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:25.819087 20060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:25.823242 20060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:25.823802 20060 catalog_manager.cc:1259] Loaded cluster ID: 82cfb4982c424cdf89384200662c34be
I20260928 21:19:25.823859 20060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 21:19:25.824687 20060 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f823ab3d9ad google::LogMessage::Fail() at ??:0
@ 0x7f823ab4088a google::LogMessage::SendToLog() at ??:0
@ 0x7f823ab3d4de google::LogMessage::Flush() at ??:0
@ 0x7f823ab3ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f824547cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f824547dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f824547a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f82454be640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f823cd8f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f823cd8fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f823cd913f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f823cd80512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f823e3c46db start_thread at ??:0
@ 0x7f823963b61f clone at ??:0
W20260928 21:19:25.905210 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:25.905503 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:25.905587 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:25.909739 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:25.909806 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:25.909897 20053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:25.915448 20053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:40407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=35239
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:25.916630 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:25.917703 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:25.927436 20093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:25.927436 20092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:25.927438 20095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:25.927824 20053 server_base.cc:1034] running on GCE node
I20260928 21:19:25.928269 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:25.928962 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:25.930162 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790630365930144 us; error 35 us; skew 500 ppm
Sep 28 21:19:25 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:25.933353 20053 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:25.934499 20053 webserver.cc:460] Webserver started at http://127.5.18.193:35239/ using document root <none> and password file <none>
I20260928 21:19:25.935094 20053 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:25.935168 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:25.939320 20053 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 21:19:25.941593 20102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:25.944049 20053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 21:19:25.944371 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "32d145f341f44bf09320076f8b43866e"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "747fd1e08e05d2a61f5065927025fd51"
server_key_iv: "95f4d9b4bf005d53d81b3b993c4d4d1e"
server_key_version: "encryptionkey@0"
I20260928 21:19:25.945180 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:25.965688 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:25.969754 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:25.969970 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:25.970659 20053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:25.971923 20109 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:19:25.976020 20053 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:19:25.976102 20053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260928 21:19:25.976174 20053 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:19:25.978271 20053 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:19:25.978319 20053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:25.978650 20109 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Bootstrap starting.
I20260928 21:19:25.988008 20053 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:40407
I20260928 21:19:25.988056 20216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:40407 every 8 connection(s)
I20260928 21:19:25.988996 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:25.991030 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20053
I20260928 21:19:25.991590 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:37991
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=42361
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:26.000651 20109 log.cc:824] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:26.010396 20109 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:26.010866 20109 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Bootstrap complete.
I20260928 21:19:26.012215 20109 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.009s
I20260928 21:19:26.015746 20109 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:26.016292 20109 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Initialized, Role: FOLLOWER
I20260928 21:19:26.016886 20109 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:26.017977 20109 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 21:19:26.097354 20219 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:19:25.991017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:56173)
0928 21:19:25.993380 (+ 2363us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:25.993448 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:25.994440 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:26.096250 (+101810us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2097,"thread_start_us":152,"threads_started":1}
W20260928 21:19:26.098014 20217 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:46147 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:46147: Client connection negotiation failed: client connection to 127.5.18.254:46147: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:19:26.098779 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.098494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:59861)
0928 21:19:26.098630 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.098708 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:19:26.099493 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.099268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:44685)
0928 21:19:26.099379 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.099431 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 21:19:26.099673 20217 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:19:26.109087 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:26.109365 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:26.109426 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:26.113500 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:26.113600 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:26.113686 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:26.119199 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:37991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=42361
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:26.120275 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:26.121306 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:26.131263 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:26.131332 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:26.131498 20220 server_base.cc:1034] running on GCE node
W20260928 21:19:26.131266 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:26.131997 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:26.132688 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:26.133848 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790630366133818 us; error 46 us; skew 500 ppm
Sep 28 21:19:26 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:26.136642 20220 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:26.137666 20220 webserver.cc:460] Webserver started at http://127.5.18.194:42361/ using document root <none> and password file <none>
I20260928 21:19:26.138262 20220 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:26.138334 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:26.142506 20220 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:19:26.145002 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:26.146430 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:19:26.146553 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "341ff820a8494250ae716265fa887817"
format_stamp: "Formatted at 2026-09-28 21:19:23 on dist-test-slave-rc9f"
server_key: "ae2f9da1968797cf35cb151b740c6a2b"
server_key_iv: "e377c3a521ef1f4ed6813cadc7057ac7"
server_key_version: "encryptionkey@0"
I20260928 21:19:26.147122 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:26.161023 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:26.164004 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:26.164189 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:26.164794 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:26.165977 20244 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:19:26.170115 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:19:26.170187 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 21:19:26.170265 20220 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:19:26.172005 20220 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:19:26.172060 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:19:26.172309 20244 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Bootstrap starting.
I20260928 21:19:26.177649 20244 log.cc:824] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:26.181526 20220 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37991
I20260928 21:19:26.181658 20352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37991 every 8 connection(s)
I20260928 21:19:26.182380 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:19:26.185352 20244 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:26.185510 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.184153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:39345)
0928 21:19:26.184630 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.184757 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260928 21:19:26.185734 20244 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Bootstrap complete.
W20260928 21:19:26.185913 20353 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:46147 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:46147: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
I20260928 21:19:26.186064 20244 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260928 21:19:26.186465 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.186267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:60945)
0928 21:19:26.186368 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.186413 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:19:26.186985 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.186799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:51191)
0928 21:19:26.186888 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.186930 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:19:26.187116 20353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:19:26.189097 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20220
I20260928 21:19:26.188813 20244 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:26.189329 20244 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341ff820a8494250ae716265fa887817, State: Initialized, Role: FOLLOWER
I20260928 21:19:26.189530 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:35815
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:46147
--webserver_port=45057
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:19:26.189822 20244 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:26.190768 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.190577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:58049)
0928 21:19:26.190669 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.190715 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 21:19:26.190820 20244 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260928 21:19:26.191427 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.191230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:59685)
0928 21:19:26.191323 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.191370 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:19:26.302745 20357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:26.303042 20357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:26.303089 20357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:26.307284 20357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:26.307363 20357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:26.307425 20357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:26.312958 20357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:35815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=45057
--tserver_master_addrs=127.5.18.254:46147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:26.314157 20357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:26.315249 20357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:26.325019 20365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:26.325030 20363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:26.325141 20362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:26.325119 20357 server_base.cc:1034] running on GCE node
I20260928 21:19:26.325750 20357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:26.326393 20357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:26.327553 20357 hybrid_clock.cc:648] HybridClock initialized: now 1790630366327530 us; error 38 us; skew 500 ppm
Sep 28 21:19:26 dist-test-slave-rc9f krb5kdc[18905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:26.330142 20357 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:26.330982 20357 webserver.cc:460] Webserver started at http://127.5.18.195:45057/ using document root <none> and password file <none>
I20260928 21:19:26.331486 20357 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:26.331539 20357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:26.335253 20357 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:26.337256 20372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:26.338554 20357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:26.338650 20357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "9c14cf0b2abd4660b5335e03cd36d0f9"
format_stamp: "Formatted at 2026-09-28 21:19:24 on dist-test-slave-rc9f"
server_key: "eead4fd1336b0c8ecb98b20a4ab4a224"
server_key_iv: "5ba793a7b88942cf88b90cbc58c165e4"
server_key_version: "encryptionkey@0"
I20260928 21:19:26.339178 20357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:26.360793 20357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:26.363992 20357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:26.364157 20357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:26.364727 20357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:26.365617 20357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:26.365669 20357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:26.365712 20357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:26.365727 20357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:26.375043 20357 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:35815
I20260928 21:19:26.375072 20485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:35815 every 8 connection(s)
I20260928 21:19:26.376102 20357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:19:26.376317 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20357
I20260928 21:19:26.380188 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.378388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:43097)
0928 21:19:26.378862 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.378974 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
W20260928 21:19:26.380725 20486 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:46147 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:46147: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
I20260928 21:19:26.381373 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.381129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:37069)
0928 21:19:26.381236 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.381312 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:19:26.381953 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:26.381773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:43407)
0928 21:19:26.381864 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:26.381905 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:19:26.382105 20486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:19:27.100844 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.100359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:38609)
0928 21:19:27.100662 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.100728 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":78,"threads_started":1}
I20260928 21:19:27.192431 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.192052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:47819)
0928 21:19:27.192284 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.192341 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260928 21:19:27.383162 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.382687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:37289)
0928 21:19:27.382969 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.383051 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260928 21:19:27.620187 20493 raft_consensus.cc:482] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:27.620381 20493 raft_consensus.cc:504] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:27.621457 20493 leader_election.cc:288] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815), 341ff820a8494250ae716265fa887817 (127.5.18.194:37991)
Sep 28 21:19:27 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.194@KRBTEST.COM
Sep 28 21:19:27 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.195@KRBTEST.COM
I20260928 21:19:27.634785 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.621716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37991 (local address 127.5.18.193:46481)
0928 21:19:27.622154 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.622168 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:27.622275 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:27.623269 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:27.623275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:27.623812 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:27.624281 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:27.624297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.625511 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.625514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:27.626493 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:27.626502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.627153 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.627157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:27.627374 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:27.628031 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:27.628042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:27.629646 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:27.631921 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:27.631926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:27.631937 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:27.632268 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:27.632653 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:27.632658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:27.632858 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:27.633604 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:27.633611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:27.633970 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 21:19:27.634214 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 21:19:27.634609 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":413,"spinlock_wait_cycles":16128,"thread_start_us":59,"threads_started":1}
I20260928 21:19:27.634831 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.621961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:46481 (local address 127.5.18.194:37991)
0928 21:19:27.622293 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:27.622310 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:27.622353 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:19:27.622601 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:27.622606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:27.622926 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:27.623198 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:27.624436 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.625378 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.626602 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.627042 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.627182 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:27.629822 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:27.629829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:27.629845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:27.629875 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:27.631797 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632405 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:27.632410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:27.632416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:27.632560 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632944 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:27.632947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:27.632949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:27.633459 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:27.633624 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:27.634334 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 21:19:27.634629 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260928 21:19:27.634786 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.621716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:35815 (local address 127.5.18.193:37289)
0928 21:19:27.622042 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.622057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:27.622160 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:27.623569 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:27.623571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:27.623825 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:27.624284 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:27.624296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.625514 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.625517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:27.626489 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:27.626498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.627153 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.627157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:27.627375 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:27.627999 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:27.628025 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:27.630062 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:27.632094 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:27.632099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:27.632308 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:27.632653 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:27.632658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:27.632858 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:27.633604 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:27.633611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:27.633970 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 21:19:27.634214 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 21:19:27.634549 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":197,"spinlock_wait_cycles":24576,"thread_start_us":84,"threads_started":1}
I20260928 21:19:27.634811 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.621863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:37289 (local address 127.5.18.195:35815)
0928 21:19:27.622374 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:27.622392 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:27.622424 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:19:27.623039 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:27.623044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:27.623366 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:27.623612 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:27.624440 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.625377 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.626598 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:27.627042 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:27.627182 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:27.630247 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:27.630252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:27.630264 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:27.630292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:27.631990 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632405 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:27.632410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:27.632416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:27.632560 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:27.632944 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:27.632947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:27.632949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:27.633459 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:27.633614 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:27.634334 (+ 720us) server_negotiation.cc:299] Negotiation successful
0928 21:19:27.634629 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":80,"threads_started":1}
I20260928 21:19:27.636149 20306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "341ff820a8494250ae716265fa887817" is_pre_election: true
I20260928 21:19:27.636448 20306 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 1.
I20260928 21:19:27.637480 20105 leader_election.cc:302] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 341ff820a8494250ae716265fa887817; no voters:
W20260928 21:19:27.637719 20103 leader_election.cc:341] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815): Not found: Tablet not found: 058408019f854f3c8d21a8ff9f8046da
I20260928 21:19:27.637802 20493 raft_consensus.cc:2781] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:19:27.637899 20493 raft_consensus.cc:482] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:27.637949 20493 raft_consensus.cc:3037] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:19:27.640235 20493 raft_consensus.cc:504] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:27.640587 20493 leader_election.cc:288] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 election: Requested vote from peers 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815), 341ff820a8494250ae716265fa887817 (127.5.18.194:37991)
I20260928 21:19:27.641093 20306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "058408019f854f3c8d21a8ff9f8046da" candidate_uuid: "32d145f341f44bf09320076f8b43866e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "341ff820a8494250ae716265fa887817"
I20260928 21:19:27.641242 20306 raft_consensus.cc:3037] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 1 FOLLOWER]: Advancing to term 2
W20260928 21:19:27.641270 20103 leader_election.cc:341] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815): Not found: Tablet not found: 058408019f854f3c8d21a8ff9f8046da
I20260928 21:19:27.643540 20306 raft_consensus.cc:2454] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d145f341f44bf09320076f8b43866e in term 2.
I20260928 21:19:27.643896 20105 leader_election.cc:302] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32d145f341f44bf09320076f8b43866e, 341ff820a8494250ae716265fa887817; no voters: 9c14cf0b2abd4660b5335e03cd36d0f9
I20260928 21:19:27.644105 20493 raft_consensus.cc:2781] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:19:27.644300 20493 raft_consensus.cc:686] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [term 2 LEADER]: Becoming Leader. State: Replica: 32d145f341f44bf09320076f8b43866e, State: Running, Role: LEADER
I20260928 21:19:27.644634 20493 consensus_queue.cc:237] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:27.645017 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:27.644750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:60777)
0928 21:19:27.644908 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:27.644954 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 21:19:28.124012 20103 consensus_peers.cc:487] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e -> Peer 9c14cf0b2abd4660b5335e03cd36d0f9 (127.5.18.195:35815): Couldn't send request to peer 9c14cf0b2abd4660b5335e03cd36d0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 058408019f854f3c8d21a8ff9f8046da. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:19:28.189015 20306 raft_consensus.cc:1261] T 058408019f854f3c8d21a8ff9f8046da P 341ff820a8494250ae716265fa887817 [term 2 FOLLOWER]: Refusing update from remote peer 32d145f341f44bf09320076f8b43866e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 21:19:28.189731 20493 consensus_queue.cc:1035] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e [LEADER]: Connected to new peer: Peer: permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 21:19:28.191666 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.191323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:33693)
0928 21:19:28.191537 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.191586 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":55,"threads_started":1}
I20260928 21:19:28.193981 20506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.193543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:50857)
0928 21:19:28.193817 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.193877 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":48,"threads_started":1}
I20260928 21:19:28.194247 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.194032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:57167)
0928 21:19:28.194144 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.194193 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:19:28.384230 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.383852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:45599)
0928 21:19:28.384070 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.384121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260928 21:19:28.681471 20508 ts_tablet_manager.cc:927] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Initiating tablet copy from peer 32d145f341f44bf09320076f8b43866e (127.5.18.193:40407)
I20260928 21:19:28.682291 20508 tablet_copy_client.cc:316] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.18.193:40407
Sep 28 21:19:28 dist-test-slave-rc9f krb5kdc[18905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:19:28.692898 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.682820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:40407 (local address 127.5.18.195:42921)
0928 21:19:28.682945 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.682969 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:28.683093 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:28.683730 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:28.683735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:28.684129 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:28.684247 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:28.684253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:28.685334 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:28.685337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:28.686088 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:28.686099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:28.686783 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:28.686787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:28.686912 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:28.687554 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:28.687567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:28.689045 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:28.691183 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:28.691194 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:28.691199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:28.691461 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:28.691733 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:28.691735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:28.691736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:28.691811 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:28.692347 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:28.692353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:28.692477 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 21:19:28.692600 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:19:28.692750 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:19:28.692950 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.682983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:42921 (local address 127.5.18.193:40407)
0928 21:19:28.683286 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:28.683297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:28.683319 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:19:28.683386 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:28.683391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:28.683579 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:28.683706 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:28.684393 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:28.685174 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:28.686220 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:28.686609 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:28.686700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:28.689224 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:28.689237 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:28.689259 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:28.689288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:28.691054 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:28.691571 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:28.691577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:28.691578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:28.691664 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:28.691928 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:28.691931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:28.691932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:28.692240 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:28.692363 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:28.692676 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:19:28.692822 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
I20260928 21:19:28.694136 20191 tablet_copy_service.cc:140] P 32d145f341f44bf09320076f8b43866e: Received BeginTabletCopySession request for tablet 058408019f854f3c8d21a8ff9f8046da from peer 9c14cf0b2abd4660b5335e03cd36d0f9 ({username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:42921)
I20260928 21:19:28.694319 20191 tablet_copy_service.cc:161] P 32d145f341f44bf09320076f8b43866e: Beginning new tablet copy session on tablet 058408019f854f3c8d21a8ff9f8046da from peer 9c14cf0b2abd4660b5335e03cd36d0f9 at {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:42921: session id = 9c14cf0b2abd4660b5335e03cd36d0f9-058408019f854f3c8d21a8ff9f8046da
I20260928 21:19:28.695520 20191 tablet_copy_source_session.cc:205] T 058408019f854f3c8d21a8ff9f8046da P 32d145f341f44bf09320076f8b43866e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:19:28.699820 20508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 058408019f854f3c8d21a8ff9f8046da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:28.706877 20508 tablet_copy_client.cc:791] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet copy: Starting download of 0 data blocks...
I20260928 21:19:28.707086 20508 tablet_copy_client.cc:655] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet copy: Starting download of 1 WAL segments...
I20260928 21:19:28.709111 20508 tablet_copy_client.cc:531] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:19:28.711100 20508 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Bootstrap starting.
I20260928 21:19:28.716795 20508 log.cc:824] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:28.723886 20508 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:19:28.724210 20508 tablet_bootstrap.cc:492] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Bootstrap complete.
I20260928 21:19:28.724535 20508 ts_tablet_manager.cc:1397] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 21:19:28.726989 20508 raft_consensus.cc:348] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:28.727265 20508 raft_consensus.cc:729] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c14cf0b2abd4660b5335e03cd36d0f9, State: Initialized, Role: FOLLOWER
I20260928 21:19:28.727712 20508 consensus_queue.cc:260] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c14cf0b2abd4660b5335e03cd36d0f9" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 35815 } } peers { permanent_uuid: "341ff820a8494250ae716265fa887817" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37991 } } peers { permanent_uuid: "32d145f341f44bf09320076f8b43866e" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 40407 } }
I20260928 21:19:28.728650 20508 ts_tablet_manager.cc:1428] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20260928 21:19:28.728914 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.728656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:36339)
0928 21:19:28.728790 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.728844 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:19:28.729359 20191 tablet_copy_service.cc:342] P 32d145f341f44bf09320076f8b43866e: Request end of tablet copy session 9c14cf0b2abd4660b5335e03cd36d0f9-058408019f854f3c8d21a8ff9f8046da received from {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:42921
I20260928 21:19:28.729472 20191 tablet_copy_service.cc:433] P 32d145f341f44bf09320076f8b43866e: ending tablet copy session 9c14cf0b2abd4660b5335e03cd36d0f9-058408019f854f3c8d21a8ff9f8046da on tablet 058408019f854f3c8d21a8ff9f8046da with peer 9c14cf0b2abd4660b5335e03cd36d0f9
I20260928 21:19:28.729557 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:28.729353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:45945)
0928 21:19:28.729452 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:28.729498 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 21:19:29.058822 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:29.058469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:42571)
0928 21:19:29.058681 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:29.058731 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":45,"threads_started":1}
I20260928 21:19:29.058897 20440 raft_consensus.cc:1203] T 058408019f854f3c8d21a8ff9f8046da P 9c14cf0b2abd4660b5335e03cd36d0f9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 21:19:29.059800 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:29.059581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:33365)
0928 21:19:29.059681 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:29.059730 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:19:29.195243 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:29.194845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:56569)
0928 21:19:29.195086 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:29.195140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260928 21:19:30.060093 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:30.059627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:57997)
0928 21:19:30.059900 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:30.059977 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260928 21:19:30.060899 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:30.060428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:48467)
0928 21:19:30.060740 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:30.060805 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":66,"threads_started":1}
I20260928 21:19:30.196388 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:30.195963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:37549)
0928 21:19:30.196229 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:30.196287 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260928 21:19:31.061501 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:31.061021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:37411)
0928 21:19:31.061313 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:31.061391 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260928 21:19:31.062023 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:31.061617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:56067)
0928 21:19:31.061898 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:31.061945 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260928 21:19:31.062222 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:31.061992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:33665)
0928 21:19:31.062107 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:31.062156 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:19:31.197800 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:31.197324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:52791)
0928 21:19:31.197634 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:31.197700 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260928 21:19:32.063386 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:32.062950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:44615)
0928 21:19:32.063196 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:32.063266 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260928 21:19:32.063489 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:32.062948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:51663)
0928 21:19:32.063314 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:32.063388 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260928 21:19:32.198912 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:32.198518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:33641)
0928 21:19:32.198754 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:32.198815 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260928 21:19:33.064630 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:33.064198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:34769)
0928 21:19:33.064461 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:33.064529 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260928 21:19:33.064670 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:33.064198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:48835)
0928 21:19:33.064534 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:33.064585 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":83,"threads_started":1}
I20260928 21:19:33.200227 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:33.199797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:36005)
0928 21:19:33.200056 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:33.200124 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260928 21:19:34.065841 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:34.065384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:42831)
0928 21:19:34.065682 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:34.065743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
I20260928 21:19:34.065865 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:34.065399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:54413)
0928 21:19:34.065728 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:34.065778 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":69,"threads_started":1}
I20260928 21:19:34.201329 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:34.200934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:54385)
0928 21:19:34.201177 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:34.201237 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260928 21:19:35.067051 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:35.066655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:47069)
0928 21:19:35.066908 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:35.066962 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":70,"threads_started":1}
I20260928 21:19:35.067051 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:35.066634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:57543)
0928 21:19:35.066894 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:35.066953 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260928 21:19:35.202422 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:35.202003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:41091)
0928 21:19:35.202249 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:35.202304 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260928 21:19:36.068245 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:36.067854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:49579)
0928 21:19:36.068098 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:36.068153 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260928 21:19:36.068271 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:36.067854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:41873)
0928 21:19:36.068141 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:36.068185 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":79,"threads_started":1}
I20260928 21:19:36.203658 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:36.203261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:50901)
0928 21:19:36.203499 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:36.203564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
I20260928 21:19:37.069617 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:37.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:56289)
0928 21:19:37.069408 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:37.069477 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260928 21:19:37.069645 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:37.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:44187)
0928 21:19:37.069467 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:37.069521 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":89,"threads_started":1}
I20260928 21:19:37.204810 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:37.204334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:50407)
0928 21:19:37.204645 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:37.204708 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":75,"threads_started":1}
I20260928 21:19:38.070900 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:38.070407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:35863)
0928 21:19:38.070762 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:38.070813 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":99,"threads_started":1}
I20260928 21:19:38.070904 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:38.070440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:60277)
0928 21:19:38.070700 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:38.070774 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260928 21:19:38.206117 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:38.205671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:44667)
0928 21:19:38.205932 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:38.205996 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260928 21:19:39.072152 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:39.071712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:36447)
0928 21:19:39.071989 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:39.072049 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260928 21:19:39.072170 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:39.071737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:41199)
0928 21:19:39.072007 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:39.072068 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260928 21:19:39.207355 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:39.206919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:56799)
0928 21:19:39.207193 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:39.207258 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260928 21:19:40.073377 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:40.072957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:46215)
0928 21:19:40.073221 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:40.073277 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260928 21:19:40.073397 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:40.072957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:55815)
0928 21:19:40.073264 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:40.073312 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":78,"threads_started":1}
I20260928 21:19:40.208837 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:40.208350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:48165)
0928 21:19:40.208670 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:40.208738 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260928 21:19:41.074594 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:41.074179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.195:46623)
0928 21:19:41.074425 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:41.074482 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260928 21:19:41.074728 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:41.074346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.193:51271)
0928 21:19:41.074578 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:41.074629 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260928 21:19:41.210309 20553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:41.209886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46147 (local address 127.5.18.194:54149)
0928 21:19:41.210147 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:41.210210 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:46147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
I20260928 21:19:41.387135 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20053
I20260928 21:19:41.394325 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20220
I20260928 21:19:41.399111 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20357
2026-09-28T21:19:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19176 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:19:41 dist-test-slave-rc9f krb5kdc[20556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:19:41 dist-test-slave-rc9f krb5kdc[20556](info): set up 2 sockets
Sep 28 21:19:41 dist-test-slave-rc9f krb5kdc[20556](info): commencing operation
krb5kdc: starting...
W20260928 21:19:42.353402 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:19:42 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:19:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:19:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:19:42.470409 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:42.580029 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:42.580446 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:42.580509 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:42.585598 20569 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:19:42.585664 20569 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:19:42.585680 20569 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:19:42.585691 20569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:19:42.585708 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:42.585726 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:42.585757 20569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:19:42.585785 20569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:19:42.591324 20569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:34467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:42.592410 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:42.593510 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:19:42.601835 20569 server_base.cc:1034] running on GCE node
W20260928 21:19:42.601821 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:42.601925 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:42.602095 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:42.602639 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:42.603749 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:42.604905 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790630382604894 us; error 32 us; skew 500 ppm
Sep 28 21:19:42 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630382, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:42.607499 20569 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:19:42.608423 20569 webserver.cc:460] Webserver started at http://127.5.18.254:40133/ using document root <none> and password file <none>
I20260928 21:19:42.608960 20569 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:42.609018 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:42.609222 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:42.611294 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "87c3db699cdc475fbeb4d32af0b47efb"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "9c83daacecd79b2ae36d027c6a9a277c"
server_key_iv: "95cf064e3da1d7e24e1da25ce4a903f0"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.611845 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "87c3db699cdc475fbeb4d32af0b47efb"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "9c83daacecd79b2ae36d027c6a9a277c"
server_key_iv: "95cf064e3da1d7e24e1da25ce4a903f0"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.615602 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:19:42.617990 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:42.619189 20569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:42.619293 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "87c3db699cdc475fbeb4d32af0b47efb"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "9c83daacecd79b2ae36d027c6a9a277c"
server_key_iv: "95cf064e3da1d7e24e1da25ce4a903f0"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.619403 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:42.661755 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:42.665009 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:42.665196 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:42.672075 20569 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:34467
I20260928 21:19:42.672336 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:34467 every 8 connection(s)
I20260928 21:19:42.673118 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:19:42.676069 20637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:42.676981 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20569
I20260928 21:19:42.677114 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:19:42.677426 5195 external_mini_cluster.cc:1428] Setting key b6a9f086c6fdb100c947285640b00d56
I20260928 21:19:42.682464 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap starting.
I20260928 21:19:42.685047 20637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:42.685961 20637 log.cc:824] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:19:42 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:42.687887 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: No bootstrap required, opened a new log
I20260928 21:19:42.691218 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:42.678814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.5.18.254:34467)
0928 21:19:42.679215 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:42.679231 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:42.679266 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:19:42.679898 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:42.679916 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:42.680321 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:42.680607 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:42.681447 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.682470 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.683576 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.684038 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.684156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:42.686901 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:42.686907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:42.686921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:42.686947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:42.688525 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:42.688928 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:42.688933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:42.688937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:42.689046 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:42.689295 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:42.689298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:42.689300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:42.689738 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:42.689920 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:42.690179 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 21:19:42.690467 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260928 21:19:42.691445 20637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:19:42.691671 20637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:42.691725 20637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Initialized, Role: FOLLOWER
I20260928 21:19:42.692137 20637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:19:42.692260 20637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:19:42.692337 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:19:42.692438 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:42.693320 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:19:42.693689 20637 leader_election.cc:302] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c3db699cdc475fbeb4d32af0b47efb; no voters:
I20260928 21:19:42.693956 20637 leader_election.cc:288] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:19:42.694068 20642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:42.694289 20642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 LEADER]: Becoming Leader. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Running, Role: LEADER
I20260928 21:19:42.694607 20642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:19:42.695228 20637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:19:42.696363 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:19:42.696377 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c3db699cdc475fbeb4d32af0b47efb. Latest consensus state: current_term: 1 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:19:42.696494 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:42.696501 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:19:42.696842 20651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:19:42.699564 20651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:19:42.705188 20651 catalog_manager.cc:1347] Generated new cluster ID: 83abcb060f4b412cad0caf20cdb4be4e
I20260928 21:19:42.705263 20651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:19:42.740484 20651 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:19:42.741845 20651 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:19:42.748857 20651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Generated new TSK 0
I20260928 21:19:42.749778 20651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:19:42.785928 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:42.896939 20663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:42.897233 20663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:42.897292 20663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:42.901403 20663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:42.901476 20663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:42.901584 20663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:19:42.907102 20663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:42.908244 20663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:42.909307 20663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:42.918751 20671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:42.918776 20668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:42.918797 20669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:42.918979 20663 server_base.cc:1034] running on GCE node
I20260928 21:19:42.919341 20663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:42.919932 20663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:42.921099 20663 hybrid_clock.cc:648] HybridClock initialized: now 1790630382921078 us; error 32 us; skew 500 ppm
Sep 28 21:19:42 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630382, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:42.923870 20663 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:19:42.924705 20663 webserver.cc:460] Webserver started at http://127.5.18.193:34503/ using document root <none> and password file <none>
I20260928 21:19:42.925231 20663 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:42.925287 20663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:42.925451 20663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:42.927465 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "51a11d28429240fdb8fec9a001aa3720"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "2ec3aa5638c8d289f82219c38c4bb041"
server_key_iv: "134e86a9a63879665dfe0192076601ed"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.927968 20663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "51a11d28429240fdb8fec9a001aa3720"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "2ec3aa5638c8d289f82219c38c4bb041"
server_key_iv: "134e86a9a63879665dfe0192076601ed"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.931856 20663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:19:42.934371 20678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:42.935595 20663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:42.935698 20663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "51a11d28429240fdb8fec9a001aa3720"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "2ec3aa5638c8d289f82219c38c4bb041"
server_key_iv: "134e86a9a63879665dfe0192076601ed"
server_key_version: "encryptionkey@0"
I20260928 21:19:42.935796 20663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:42.954967 20663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:42.958262 20663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:42.958421 20663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:42.958976 20663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:42.959792 20663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:42.959838 20663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:42.959877 20663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:42.959897 20663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:42.968385 20663 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45745
I20260928 21:19:42.968420 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45745 every 8 connection(s)
I20260928 21:19:42.969303 20663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:19:42.972407 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20663
I20260928 21:19:42.972565 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:19:42.972913 5195 external_mini_cluster.cc:1428] Setting key 04e9807c12e2f8a3d20833e9a6619a6b
Sep 28 21:19:42 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630382, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:19:42.983728 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:42.971127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:57281 (local address 127.5.18.254:34467)
0928 21:19:42.971256 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:42.971260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:42.972172 (+ 912us) server_negotiation.cc:407] Connection header received
0928 21:19:42.973781 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:42.973785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:42.973845 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:42.973913 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:42.975465 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.976112 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.977292 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.977549 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.977664 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:42.980268 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:42.980271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:42.980273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:42.980300 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:42.981698 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:42.982232 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:42.982235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:42.982238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:42.982316 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:42.982637 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:42.982640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:42.982642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:42.982846 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:42.982941 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:42.983465 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 21:19:42.983589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:19:42.984702 20794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:42.971543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:57281)
0928 21:19:42.971987 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:42.972033 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:42.973500 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:42.974124 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:42.974131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:42.974568 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:42.975291 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:42.975311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.976242 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.976246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:42.977171 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:42.977180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:42.977728 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:42.977732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:42.977944 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:42.978587 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:42.978605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:42.980089 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:42.981804 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:42.981810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:42.981829 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:42.982137 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:42.982398 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:42.982401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:42.982404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:42.982550 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:42.982956 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:42.982961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:42.983221 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 21:19:42.983431 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 21:19:42.983737 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260928 21:19:42.986074 20792 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:19:42.986348 20792 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:42.986954 20792 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:19:42.988643 20601 ts_manager.cc:194] Registered new tserver with Master: 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:19:42.990254 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:57281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:19:43.000856 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:43.110072 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:43.110365 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:43.110423 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:43.114502 20797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:43.114564 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:43.114641 20797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:19:43.121186 20797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:43.122720 20797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:43.123909 20797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:43.133345 20803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:43.133407 20797 server_base.cc:1034] running on GCE node
W20260928 21:19:43.133423 20805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:43.133469 20802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:43.134083 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:43.134743 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:43.135928 20797 hybrid_clock.cc:648] HybridClock initialized: now 1790630383135908 us; error 31 us; skew 500 ppm
Sep 28 21:19:43 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630383, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:43.138829 20797 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:19:43.139849 20797 webserver.cc:460] Webserver started at http://127.5.18.194:36589/ using document root <none> and password file <none>
I20260928 21:19:43.140427 20797 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:43.140501 20797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:43.140724 20797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:43.142845 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "5bb8cb671b69f1fa77ce39f9ef3811bf"
server_key_iv: "d45cb1648dd89d5b529d9da91129c2ad"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.143404 20797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "5bb8cb671b69f1fa77ce39f9ef3811bf"
server_key_iv: "d45cb1648dd89d5b529d9da91129c2ad"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.147156 20797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:19:43.149717 20812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.150965 20797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:19:43.151077 20797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "5bb8cb671b69f1fa77ce39f9ef3811bf"
server_key_iv: "d45cb1648dd89d5b529d9da91129c2ad"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.151177 20797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:43.179198 20797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:43.182219 20797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:43.182403 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:43.182994 20797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:43.183817 20797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:43.183864 20797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.183917 20797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:43.183941 20797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.192386 20797 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37329
I20260928 21:19:43.192466 20925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37329 every 8 connection(s)
I20260928 21:19:43.193274 20797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:19:43.197180 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20797
I20260928 21:19:43.197276 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:19:43.197533 5195 external_mini_cluster.cc:1428] Setting key 7192e14d3143dbd05de413d3c5123b95
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Sep 28 21:19:43 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630383, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:43.213649 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.195072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:54179 (local address 127.5.18.254:34467)
0928 21:19:43.195217 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.195221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.196125 (+ 904us) server_negotiation.cc:407] Connection header received
0928 21:19:43.197697 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.197701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.197761 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.197838 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:43.199273 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.199800 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.201410 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.201752 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.201858 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.210062 (+ 8204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:43.210066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:43.210069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:43.210100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:43.211497 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.212055 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.212058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.212059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.212139 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.212484 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.212487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.212488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.212684 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:43.212771 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.213318 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.213457 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:19:43.214537 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.195437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:54179)
0928 21:19:43.195940 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.195999 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.197440 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.198022 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.198030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.198505 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:43.199136 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.199157 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.199925 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.199929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.201261 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.201279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.201907 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.201910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.202072 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.205923 (+ 3851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:43.205943 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:43.209820 (+ 3877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:43.211631 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:43.211638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:43.211658 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:43.211939 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:43.212226 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:43.212229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:43.212231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:43.212388 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:43.212785 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:43.212793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:43.213077 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.213258 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.213648 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":141,"threads_started":1}
I20260928 21:19:43.215709 20926 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:19:43.215962 20926 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:43.216468 20926 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:19:43.217515 20601 ts_manager.cc:194] Registered new tserver with Master: c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
I20260928 21:19:43.218186 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:54179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:19:43.224691 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:19:43.336705 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:19:43.337000 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:19:43.337050 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:19:43.341323 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:19:43.341398 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:19:43.341462 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:19:43.346967 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:19:43.348064 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:19:43.349073 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:19:43.358960 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:19:43.358966 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:43.359197 20931 server_base.cc:1034] running on GCE node
W20260928 21:19:43.358960 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:19:43.359685 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:19:43.360383 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:19:43.361589 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790630383361538 us; error 44 us; skew 500 ppm
Sep 28 21:19:43 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630383, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:19:43.364441 20931 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:19:43.365360 20931 webserver.cc:460] Webserver started at http://127.5.18.195:33579/ using document root <none> and password file <none>
I20260928 21:19:43.365972 20931 fs_manager.cc:359] Metadata directory not provided
I20260928 21:19:43.366043 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:19:43.366252 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:19:43.368367 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "11051de5271e44df8a1c2e8a57c68ee7"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "7e0b8206f137c03d4dacc652d84f0077"
server_key_iv: "a0a0046b9ba8c17da9ffd0e5421a591f"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.368973 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "11051de5271e44df8a1c2e8a57c68ee7"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "7e0b8206f137c03d4dacc652d84f0077"
server_key_iv: "a0a0046b9ba8c17da9ffd0e5421a591f"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.372771 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:19:43.375378 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.376627 20931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:19:43.376744 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "11051de5271e44df8a1c2e8a57c68ee7"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "7e0b8206f137c03d4dacc652d84f0077"
server_key_iv: "a0a0046b9ba8c17da9ffd0e5421a591f"
server_key_version: "encryptionkey@0"
I20260928 21:19:43.376863 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:19:43.389094 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:19:43.392203 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:19:43.392400 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:19:43.392990 20931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:19:43.393975 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:19:43.394027 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.394099 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:19:43.394126 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:19:43.402976 20931 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:39807
I20260928 21:19:43.403003 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:39807 every 8 connection(s)
I20260928 21:19:43.403884 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:19:43.410866 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 20931
I20260928 21:19:43.410987 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:19:43.411310 5195 external_mini_cluster.cc:1428] Setting key 5421a82cdb1dea176786ec78f2652a5d
Sep 28 21:19:43 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630383, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:19:43.416781 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.405684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:50205 (local address 127.5.18.254:34467)
0928 21:19:43.405841 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.405846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.406750 (+ 904us) server_negotiation.cc:407] Connection header received
0928 21:19:43.408002 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.408004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.408054 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.408125 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:43.409270 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.409831 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.410952 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.411263 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.411371 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.413967 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:43.413971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:43.413973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:43.413998 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:43.415103 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.415557 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.415559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.415560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.415623 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.415865 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.415867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.415868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.416026 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:43.416096 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.416566 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.416663 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:19:43.417649 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.406107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:50205)
0928 21:19:43.406593 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.406638 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.407803 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.408284 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.408290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.408658 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:19:43.409156 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.409169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.409935 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.409937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.410834 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.410843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.411414 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.411419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.411573 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.412102 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:19:43.412123 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:19:43.413808 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:19:43.415195 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:43.415200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:43.415213 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:43.415478 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:43.415681 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:19:43.415683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:19:43.415685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:19:43.415808 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:19:43.416110 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:19:43.416115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:19:43.416352 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.416529 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.416783 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260928 21:19:43.418685 21060 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:19:43.418912 21060 heartbeater.cc:464] Registering TS with master...
I20260928 21:19:43.419379 21060 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:19:43.420359 20601 ts_manager.cc:194] Registered new tserver with Master: 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807)
I20260928 21:19:43.420922 20601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:50205
I20260928 21:19:43.425017 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:19:43.434546 20640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.426896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.5.18.254:34467)
0928 21:19:43.427025 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.427028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.427172 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:19:43.427274 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.427276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.427315 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.427380 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:19:43.428008 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.428521 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.429345 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.429651 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.429763 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.431263 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:19:43.431268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:19:43.431272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:19:43.431308 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:19:43.432678 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.433148 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.433153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.433155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.433245 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:19:43.433518 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:19:43.433521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:19:43.433523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:19:43.433873 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:19:43.434036 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.434260 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.434412 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:19:43.437405 20601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:19:43.440619 20601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:19:43.455271 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.449074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.5.18.194:37329)
0928 21:19:43.449929 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.449939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.449964 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:19:43.450493 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.450498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.450672 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.450809 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:43.451261 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452075 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.453903 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.454586 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.454682 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.454841 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.454999 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.455108 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":127,"threads_started":1}
I20260928 21:19:43.455278 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.448981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.0.0.1:36042)
0928 21:19:43.449802 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.449856 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.449989 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.451089 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.451092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.451105 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:43.451163 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.451167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452187 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.452190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.453789 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.453796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.454696 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.454699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.454808 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.454843 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.454954 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.455151 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":147,"thread_start_us":92,"threads_started":1}
I20260928 21:19:43.455549 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.449774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.0.0.1:55428)
0928 21:19:43.450216 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.450228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.450312 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.451433 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.451436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.451454 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:43.451542 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.451548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452789 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.452792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.453795 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.453805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.455052 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.455055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.455238 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.455274 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.455327 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.455409 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":133,"spinlock_wait_cycles":896,"thread_start_us":88,"threads_started":1}
I20260928 21:19:43.455610 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.449275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.5.18.195:39807)
0928 21:19:43.449739 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.449751 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.450420 (+ 669us) server_negotiation.cc:407] Connection header received
0928 21:19:43.450503 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.450507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.450672 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.450789 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:43.451100 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452244 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.453898 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.454731 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.454837 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.454989 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.455390 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.455484 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":115,"threads_started":1}
I20260928 21:19:43.456059 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.450161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.5.18.193:45745)
0928 21:19:43.451019 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.451030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.451049 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:19:43.451118 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.451123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.451296 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.451448 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:43.451645 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452664 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.453923 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.454790 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.454871 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.454989 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.455822 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.455919 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":119,"threads_started":1}
I20260928 21:19:43.456099 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.448999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39807 (local address 127.0.0.1:57890)
0928 21:19:43.450197 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.450210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.450310 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.450816 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.450822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.450843 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:43.450978 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.450985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.452372 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.452375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.453762 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.453773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.455052 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.455055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.455206 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.455262 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.455920 (+ 658us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.455988 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":79,"spinlock_wait_cycles":10752,"thread_start_us":86,"threads_started":1}
I20260928 21:19:43.458748 20726 tablet_service.cc:1467] Processing CreateTablet for tablet 4877323293ce483ea06d2032d2e16516 (DEFAULT_TABLE table=test-table [id=8d0231af6e384814a32b7e189af0196f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:43.458865 20994 tablet_service.cc:1467] Processing CreateTablet for tablet 4877323293ce483ea06d2032d2e16516 (DEFAULT_TABLE table=test-table [id=8d0231af6e384814a32b7e189af0196f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:43.458866 20860 tablet_service.cc:1467] Processing CreateTablet for tablet 4877323293ce483ea06d2032d2e16516 (DEFAULT_TABLE table=test-table [id=8d0231af6e384814a32b7e189af0196f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:19:43.459779 20726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4877323293ce483ea06d2032d2e16516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:43.459878 20860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4877323293ce483ea06d2032d2e16516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:43.459892 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4877323293ce483ea06d2032d2e16516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:19:43.466518 21077 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap starting.
I20260928 21:19:43.468968 21077 tablet_bootstrap.cc:654] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:43.469956 21078 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Bootstrap starting.
I20260928 21:19:43.470211 21077 log.cc:824] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:43.470551 21079 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Bootstrap starting.
I20260928 21:19:43.472644 21078 tablet_bootstrap.cc:654] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:43.473008 21079 tablet_bootstrap.cc:654] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Neither blocks nor log segments found. Creating new log.
I20260928 21:19:43.473150 21077 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: No bootstrap required, opened a new log
I20260928 21:19:43.473362 21077 ts_tablet_manager.cc:1397] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:19:43.473520 21078 log.cc:824] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:43.473831 21079 log.cc:824] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Log is configured to *not* fsync() on all Append() calls
I20260928 21:19:43.475818 21079 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: No bootstrap required, opened a new log
I20260928 21:19:43.475837 21078 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: No bootstrap required, opened a new log
I20260928 21:19:43.476035 21078 ts_tablet_manager.cc:1397] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:19:43.476035 21079 ts_tablet_manager.cc:1397] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:19:43.476728 21077 raft_consensus.cc:348] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.476938 21077 raft_consensus.cc:374] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:43.476967 21077 raft_consensus.cc:729] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a11d28429240fdb8fec9a001aa3720, State: Initialized, Role: FOLLOWER
I20260928 21:19:43.477356 21077 consensus_queue.cc:260] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.478003 20792 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
I20260928 21:19:43.479457 21077 ts_tablet_manager.cc:1428] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 21:19:43.480119 21078 raft_consensus.cc:348] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.480119 21079 raft_consensus.cc:348] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.480439 21079 raft_consensus.cc:374] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:43.480481 21078 raft_consensus.cc:374] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:19:43.480541 21079 raft_consensus.cc:729] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7af4d109c6c4c68936a2ca5a6257d56, State: Initialized, Role: FOLLOWER
I20260928 21:19:43.480589 21078 raft_consensus.cc:729] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11051de5271e44df8a1c2e8a57c68ee7, State: Initialized, Role: FOLLOWER
I20260928 21:19:43.481076 21079 consensus_queue.cc:260] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.481076 21078 consensus_queue.cc:260] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.481853 20926 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
I20260928 21:19:43.482051 21060 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
I20260928 21:19:43.482239 21079 ts_tablet_manager.cc:1428] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:19:43.482239 21078 ts_tablet_manager.cc:1428] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 21:19:43.654870 21061 tablet.cc:2369] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:19:43.694379 20927 tablet.cc:2369] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:19:43.720674 20793 tablet.cc:2369] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:19:43.788156 21084 raft_consensus.cc:482] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:19:43.788367 21084 raft_consensus.cc:504] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.789423 21084 leader_election.cc:288] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:19:43.793515 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.790236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:33597 (local address 127.5.18.193:45745)
0928 21:19:43.790352 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.790356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.790435 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:19:43.790594 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.790597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.790641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.790703 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:43.790975 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.791455 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.792551 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.792938 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.793049 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.793101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.793304 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.793378 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:19:43.793524 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.789823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:44311 (local address 127.5.18.195:39807)
0928 21:19:43.789968 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.789971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.790078 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:19:43.790271 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.790275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.790404 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.790484 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:19:43.790910 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.791428 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.792545 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.792939 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.793044 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.793101 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.793317 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.793398 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:19:43.793547 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.790112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.5.18.194:33597)
0928 21:19:43.790351 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.790364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.790490 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.790724 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.790727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.790742 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:43.790859 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.790865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.791565 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.791568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.792424 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.792434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.793051 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.793054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.793185 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.793215 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.793277 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.793387 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":64,"thread_start_us":67,"threads_started":1}
I20260928 21:19:43.793548 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.789730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39807 (local address 127.5.18.194:44311)
0928 21:19:43.789992 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.790006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:19:43.790124 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:19:43.790615 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:19:43.790617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:19:43.790633 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:19:43.790729 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.790733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.791539 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.791542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.792421 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:19:43.792430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.793044 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.793048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:19:43.793197 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.793225 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:19:43.793287 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:19:43.793394 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":64,"thread_start_us":98,"threads_started":1}
I20260928 21:19:43.794286 21014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4877323293ce483ea06d2032d2e16516" candidate_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11051de5271e44df8a1c2e8a57c68ee7" is_pre_election: true
I20260928 21:19:43.794288 20746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4877323293ce483ea06d2032d2e16516" candidate_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a11d28429240fdb8fec9a001aa3720" is_pre_election: true
I20260928 21:19:43.794564 20746 raft_consensus.cc:2454] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7af4d109c6c4c68936a2ca5a6257d56 in term 0.
I20260928 21:19:43.794569 21014 raft_consensus.cc:2454] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7af4d109c6c4c68936a2ca5a6257d56 in term 0.
I20260928 21:19:43.795017 20813 leader_election.cc:302] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56; no voters:
I20260928 21:19:43.795262 21084 raft_consensus.cc:2781] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:19:43.795336 21084 raft_consensus.cc:482] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:19:43.795365 21084 raft_consensus.cc:3037] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:43.796470 21084 raft_consensus.cc:504] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.796788 21084 leader_election.cc:288] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [CANDIDATE]: Term 1 election: Requested vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:19:43.797112 21014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4877323293ce483ea06d2032d2e16516" candidate_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11051de5271e44df8a1c2e8a57c68ee7"
I20260928 21:19:43.797152 20746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4877323293ce483ea06d2032d2e16516" candidate_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a11d28429240fdb8fec9a001aa3720"
I20260928 21:19:43.797284 20746 raft_consensus.cc:3037] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:43.797374 21014 raft_consensus.cc:3037] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:19:43.798328 20746 raft_consensus.cc:2454] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7af4d109c6c4c68936a2ca5a6257d56 in term 1.
I20260928 21:19:43.798367 21014 raft_consensus.cc:2454] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7af4d109c6c4c68936a2ca5a6257d56 in term 1.
I20260928 21:19:43.798638 20813 leader_election.cc:302] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56; no voters:
I20260928 21:19:43.798796 21084 raft_consensus.cc:2781] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:19:43.799007 21084 raft_consensus.cc:686] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 LEADER]: Becoming Leader. State: Replica: c7af4d109c6c4c68936a2ca5a6257d56, State: Running, Role: LEADER
I20260928 21:19:43.799358 21084 consensus_queue.cc:237] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:19:43.802415 20600 catalog_manager.cc:5495] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } health_report { overall_health: UNKNOWN } } }
I20260928 21:19:43.818214 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.814794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.5.18.194:37329)
0928 21:19:43.814919 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:19:43.814923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:19:43.814937 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:19:43.815041 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:19:43.815044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:19:43.815099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:19:43.815197 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:19:43.815501 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.815994 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.816864 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:19:43.817084 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:19:43.817161 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:19:43.817389 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:19:43.817877 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:19:43.817967 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:19:43.818038 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 21:19:43.818095 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:19:43.824419 21014 raft_consensus.cc:1261] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Refusing update from remote peer c7af4d109c6c4c68936a2ca5a6257d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:43.824518 20746 raft_consensus.cc:1261] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Refusing update from remote peer c7af4d109c6c4c68936a2ca5a6257d56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:19:43.825207 21088 consensus_queue.cc:1035] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:43.825444 21084 consensus_queue.cc:1035] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:19:43.832939 21091 mvcc.cc:205] Tried to move back new op lower bound from 7334422052137054208 to 7334422052044972032. Current Snapshot: MvccSnapshot[applied={T|T < 7334422052137054208}]
I20260928 21:19:43.834050 21093 mvcc.cc:205] Tried to move back new op lower bound from 7334422052137054208 to 7334422052044972032. Current Snapshot: MvccSnapshot[applied={T|T < 7334422052137054208}]
I20260928 21:19:43.844105 20599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:43.844345 20599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:19:43.847324 20599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Sending DeleteTablet for 3 replicas of tablet 4877323293ce483ea06d2032d2e16516
I20260928 21:19:43.848282 20860 tablet_service.cc:1514] Processing DeleteTablet for tablet 4877323293ce483ea06d2032d2e16516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:36042
I20260928 21:19:43.848366 20994 tablet_service.cc:1514] Processing DeleteTablet for tablet 4877323293ce483ea06d2032d2e16516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:57890
I20260928 21:19:43.848559 20726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4877323293ce483ea06d2032d2e16516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:55428
I20260928 21:19:43.848934 21101 tablet_replica.cc:330] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: stopping tablet replica
I20260928 21:19:43.849030 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20663
I20260928 21:19:43.849849 21101 raft_consensus.cc:2229] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:19:43.850268 21101 raft_consensus.cc:2258] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:43.850831 21102 tablet_replica.cc:330] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: stopping tablet replica
I20260928 21:19:43.851171 21102 raft_consensus.cc:2229] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:19:43.851531 21102 raft_consensus.cc:2258] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:19:43.852078 21101 ts_tablet_manager.cc:1905] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:43.853048 21102 ts_tablet_manager.cc:1905] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:19:43.854988 21101 ts_tablet_manager.cc:1918] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:43.855237 21101 log.cc:1196] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/4877323293ce483ea06d2032d2e16516
I20260928 21:19:43.855654 21101 ts_tablet_manager.cc:1939] T 4877323293ce483ea06d2032d2e16516 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting consensus metadata
W20260928 21:19:43.855733 20585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:45745 (error 108)
I20260928 21:19:43.855767 21102 ts_tablet_manager.cc:1918] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:19:43.855852 21102 log.cc:1196] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/4877323293ce483ea06d2032d2e16516
I20260928 21:19:43.856154 21102 ts_tablet_manager.cc:1939] T 4877323293ce483ea06d2032d2e16516 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting consensus metadata
I20260928 21:19:43.856571 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20797
I20260928 21:19:43.858588 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.858232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.0.0.1:56496)
0928 21:19:43.858399 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.858491 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:19:43.858779 20585 catalog_manager.cc:4584] TS 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4877323293ce483ea06d2032d2e16516: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45745: connect: Connection refused (error 111)
I20260928 21:19:43.859021 20585 catalog_manager.cc:4841] TS 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): tablet 4877323293ce483ea06d2032d2e16516 (table test-table [id=8d0231af6e384814a32b7e189af0196f]) successfully deleted
I20260928 21:19:43.861999 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20931
I20260928 21:19:43.863351 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:19:43.862245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.0.0.1:48496)
0928 21:19:43.863190 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 21:19:43.863252 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":851}
I20260928 21:19:43.867072 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 20569
I20260928 21:20:03.873131 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--webserver_interface=127.5.18.254
--webserver_port=40133
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:03.983449 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:03.983755 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:03.983810 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:03.987921 21105 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:20:03.987987 21105 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:20:03.988011 21105 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:20:03.988029 21105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:03.988054 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:03.988085 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:03.988116 21105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:03.988143 21105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:03.993582 21105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:34467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=40133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:03.994676 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:03.995682 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:04.008589 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.008615 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.008565 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:04.008551 21105 server_base.cc:1034] running on GCE node
I20260928 21:20:04.009615 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:04.010895 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:04.012077 21105 hybrid_clock.cc:648] HybridClock initialized: now 1790630404012059 us; error 38 us; skew 500 ppm
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:04.018730 21105 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:20:04.019786 21105 webserver.cc:460] Webserver started at http://127.5.18.254:40133/ using document root <none> and password file <none>
I20260928 21:20:04.020398 21105 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:04.020486 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:04.024464 21105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:20:04.026646 21121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.027925 21105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:20:04.028056 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "87c3db699cdc475fbeb4d32af0b47efb"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "9c83daacecd79b2ae36d027c6a9a277c"
server_key_iv: "95cf064e3da1d7e24e1da25ce4a903f0"
server_key_version: "encryptionkey@0"
I20260928 21:20:04.028649 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:04.048843 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:04.052126 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:04.052317 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:04.059336 21105 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:34467
I20260928 21:20:04.059595 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:34467 every 8 connection(s)
I20260928 21:20:04.060462 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:04.064965 21174 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:20:04.066877 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap starting.
I20260928 21:20:04.070204 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21105
I20260928 21:20:04.070725 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:45745
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=34503
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:20:04.084514 21174 log.cc:824] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:04.093148 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:20:04.093500 21174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap complete.
I20260928 21:20:04.097400 21174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:04.097875 21174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Initialized, Role: FOLLOWER
I20260928 21:20:04.098397 21174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:04.098523 21174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:04.098596 21174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:04.098692 21174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:20:04.100659 21174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:04.100984 21174 leader_election.cc:302] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c3db699cdc475fbeb4d32af0b47efb; no voters:
I20260928 21:20:04.101282 21174 leader_election.cc:288] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:20:04.101419 21178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:20:04.101697 21178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 LEADER]: Becoming Leader. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Running, Role: LEADER
I20260928 21:20:04.102039 21178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:04.102494 21174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:04.103426 21179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:20:04.103567 21179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:04.103937 21180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c3db699cdc475fbeb4d32af0b47efb. Latest consensus state: current_term: 2 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:20:04.104074 21180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:04.104460 21187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:04.108559 21187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:04.109054 21187 catalog_manager.cc:1259] Loaded cluster ID: 83abcb060f4b412cad0caf20cdb4be4e
I20260928 21:20:04.109103 21187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:04.110828 21187 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:04.111403 21187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Loaded TSK: 0
I20260928 21:20:04.123654 21187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Generated new TSK 1
I20260928 21:20:04.123906 21187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:20:04.198431 21176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:04.198725 21176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:04.198788 21176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:04.203028 21176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:04.203104 21176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:04.203186 21176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:04.208878 21176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:45745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=34503
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:04.209933 21176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:04.210923 21176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:04.221376 21201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.221385 21202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:04.221552 21176 server_base.cc:1034] running on GCE node
W20260928 21:20:04.221736 21204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:04.222239 21176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:04.223078 21176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:04.224260 21176 hybrid_clock.cc:648] HybridClock initialized: now 1790630404224229 us; error 46 us; skew 500 ppm
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:04.227674 21176 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:20:04.228619 21176 webserver.cc:460] Webserver started at http://127.5.18.193:34503/ using document root <none> and password file <none>
I20260928 21:20:04.229173 21176 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:04.229247 21176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:04.233371 21176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:20:04.235703 21211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.237061 21176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 21:20:04.237167 21176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "51a11d28429240fdb8fec9a001aa3720"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "2ec3aa5638c8d289f82219c38c4bb041"
server_key_iv: "134e86a9a63879665dfe0192076601ed"
server_key_version: "encryptionkey@0"
I20260928 21:20:04.237751 21176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:04.269008 21176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:04.271924 21176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:04.272109 21176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:04.272711 21176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:04.273772 21218 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:20:04.277776 21176 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:20:04.277839 21176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 21:20:04.277901 21176 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:20:04.279546 21176 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:20:04.279592 21176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:20:04.279852 21218 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap starting.
I20260928 21:20:04.289088 21176 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45745
I20260928 21:20:04.289152 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45745 every 8 connection(s)
I20260928 21:20:04.290136 21176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:04.297420 21218 log.cc:824] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:04.298256 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21176
I20260928 21:20:04.298741 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:37329
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=36589
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:04.306090 21218 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:20:04.306238 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.292219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56853 (local address 127.5.18.254:34467)
0928 21:20:04.292560 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.292581 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.292943 (+ 362us) server_negotiation.cc:407] Connection header received
0928 21:20:04.294081 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.294092 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.294408 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.294714 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.295848 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.296814 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.298008 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.298526 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.298694 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.301142 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.301148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.301165 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.301197 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.302963 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.303437 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.303443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.303447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.303582 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.303990 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.303994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.303995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.304439 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.304531 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.305116 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.305382 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":93,"threads_started":1}
I20260928 21:20:04.306257 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.292296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:56853)
0928 21:20:04.292779 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.292823 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.293671 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.294785 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.294792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.295222 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:04.295679 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.295692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.296945 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.296948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.297895 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.297905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.298655 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.298659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.298841 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.299437 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:04.299453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:04.300937 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:04.303076 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.303081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.303092 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.303333 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.303700 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.303703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.303705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.303883 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.304607 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:04.304611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:04.304827 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.304980 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.305309 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":134,"threads_started":1}
I20260928 21:20:04.306531 21218 tablet_bootstrap.cc:492] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap complete.
I20260928 21:20:04.307875 21218 ts_tablet_manager.cc:1397] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20260928 21:20:04.308367 21326 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:20:04.308589 21326 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:04.309054 21326 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:20:04.311349 21138 ts_manager.cc:194] Registered new tserver with Master: 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:20:04.311089 21218 raft_consensus.cc:348] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.311612 21218 raft_consensus.cc:729] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a11d28429240fdb8fec9a001aa3720, State: Initialized, Role: FOLLOWER
I20260928 21:20:04.312150 21218 consensus_queue.cc:260] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.313180 21218 ts_tablet_manager.cc:1428] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:20:04.315065 21138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:56853
I20260928 21:20:04.317667 21326 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.193@KRBTEST.COM
I20260928 21:20:04.325434 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.314453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.0.0.1:54472)
0928 21:20:04.315197 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.315250 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.315400 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.315973 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.315979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.316424 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:04.316558 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.316567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.317870 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.317874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.318890 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.318901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.319689 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.319692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.319842 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.320373 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:04.320385 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:04.321827 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:04.323870 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.323879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.323884 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.324101 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.324385 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.324387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.324388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.324460 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.324919 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:04.324924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:04.325029 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.325137 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.325299 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":108,"threads_started":1}
I20260928 21:20:04.325595 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.314544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.5.18.193:45745)
0928 21:20:04.314925 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.314939 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.315352 (+ 413us) server_negotiation.cc:407] Connection header received
0928 21:20:04.315562 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.315568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.315775 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.315913 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.316692 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.317730 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.319037 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.319559 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.319655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.321977 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.321989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.322004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.322038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.323774 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.324203 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.324209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.324211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.324304 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.324528 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.324531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.324532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.324826 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.324948 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.325230 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.325401 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
I20260928 21:20:04.326697 21260 tablet_service.cc:1514] Processing DeleteTablet for tablet 4877323293ce483ea06d2032d2e16516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:19:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:54472
I20260928 21:20:04.327498 21335 tablet_replica.cc:330] stopping tablet replica
I20260928 21:20:04.327800 21335 raft_consensus.cc:2229] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:04.328040 21335 raft_consensus.cc:2258] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:04.328964 21335 ts_tablet_manager.cc:1905] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:04.332789 21335 ts_tablet_manager.cc:1918] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:04.332878 21335 log.cc:1196] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal/wals/4877323293ce483ea06d2032d2e16516
I20260928 21:20:04.333102 21335 ts_tablet_manager.cc:1939] T 4877323293ce483ea06d2032d2e16516 P 51a11d28429240fdb8fec9a001aa3720: Deleting consensus metadata
I20260928 21:20:04.334870 21122 catalog_manager.cc:4841] TS 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745): tablet 4877323293ce483ea06d2032d2e16516 (table test-table [id=8d0231af6e384814a32b7e189af0196f]) successfully deleted
W20260928 21:20:04.415910 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:04.416193 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:04.416250 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:04.420305 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:04.420373 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:04.420463 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:04.425807 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:37329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=36589
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:04.426949 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:04.427963 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:20:04.437466 21331 server_base.cc:1034] running on GCE node
W20260928 21:20:04.437476 21340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.437413 21341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.437417 21343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:04.438082 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:04.438678 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:04.439826 21331 hybrid_clock.cc:648] HybridClock initialized: now 1790630404439809 us; error 35 us; skew 500 ppm
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:04.442559 21331 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:20:04.443410 21331 webserver.cc:460] Webserver started at http://127.5.18.194:36589/ using document root <none> and password file <none>
I20260928 21:20:04.443929 21331 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:04.444018 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:04.447752 21331 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:04.449880 21350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.451107 21331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:20:04.451215 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "5bb8cb671b69f1fa77ce39f9ef3811bf"
server_key_iv: "d45cb1648dd89d5b529d9da91129c2ad"
server_key_version: "encryptionkey@0"
I20260928 21:20:04.451756 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:04.469295 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:04.472391 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:04.472577 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:04.473169 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:04.474028 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:04.474076 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.474126 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:04.474166 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.482904 21331 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37329
I20260928 21:20:04.482932 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37329 every 8 connection(s)
I20260928 21:20:04.483911 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:04.486419 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21331
I20260928 21:20:04.486820 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:39807
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=33579
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:04.497289 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.485772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:46775 (local address 127.5.18.254:34467)
0928 21:20:04.485939 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.485943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.486810 (+ 867us) server_negotiation.cc:407] Connection header received
0928 21:20:04.488096 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.488099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.488158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.488227 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.489398 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.489971 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.491099 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.491338 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.491424 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.493766 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.493769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.493771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.493794 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.495253 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.495799 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.495802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.495803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.495878 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.496193 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.496196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.496198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.496389 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.496476 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.496983 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.497123 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:20:04.498159 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.486175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:46775)
0928 21:20:04.486650 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.486700 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.487891 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.488418 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.488425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.488787 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:04.489258 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.489272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.490088 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.490091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.490986 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.490995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.491482 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.491486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.491645 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.492184 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:04.492200 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:04.493611 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:04.495378 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.495385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.495401 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.495699 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.495962 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.495965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.495967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.496109 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.496489 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:04.496496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:04.496750 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.496920 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.497253 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
I20260928 21:20:04.499327 21464 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:20:04.499555 21464 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:04.500066 21464 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:20:04.501078 21138 ts_manager.cc:194] Registered new tserver with Master: c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
I20260928 21:20:04.501848 21138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:46775
W20260928 21:20:04.597301 21467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:04.597627 21467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:04.597682 21467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:04.601701 21467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:04.601773 21467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:04.601853 21467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:04.607386 21467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:39807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=33579
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:04.608417 21467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:04.609356 21467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:04.618426 21472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.618538 21475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:04.618561 21473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:04.618716 21467 server_base.cc:1034] running on GCE node
I20260928 21:20:04.619122 21467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:04.619764 21467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:04.620922 21467 hybrid_clock.cc:648] HybridClock initialized: now 1790630404620906 us; error 30 us; skew 500 ppm
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:04.623685 21467 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:20:04.624531 21467 webserver.cc:460] Webserver started at http://127.5.18.195:33579/ using document root <none> and password file <none>
I20260928 21:20:04.625061 21467 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:04.625139 21467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:04.629038 21467 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:04.631198 21482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.632484 21467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:20:04.632602 21467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "11051de5271e44df8a1c2e8a57c68ee7"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "7e0b8206f137c03d4dacc652d84f0077"
server_key_iv: "a0a0046b9ba8c17da9ffd0e5421a591f"
server_key_version: "encryptionkey@0"
I20260928 21:20:04.633152 21467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:04.656086 21467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:04.659086 21467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:04.659261 21467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:04.659847 21467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:04.660669 21467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:04.660717 21467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.660766 21467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:04.660794 21467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:04.669226 21467 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:39807
I20260928 21:20:04.669253 21595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:39807 every 8 connection(s)
I20260928 21:20:04.670249 21467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:04.673125 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21467
Sep 28 21:20:04 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630404, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:04.682798 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.673831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.5.18.254:34467)
0928 21:20:04.674092 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.674097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.674310 (+ 213us) server_negotiation.cc:407] Connection header received
0928 21:20:04.674445 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.674448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.674496 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.674574 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.675335 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.676078 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.676996 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.677228 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.677337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.679080 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.679084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.679087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.679115 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.680638 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.681276 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.681279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.681280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.681376 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.681726 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.681729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.681731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.681997 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.682175 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.682451 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.682616 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260928 21:20:04.685915 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.672312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:40621 (local address 127.5.18.254:34467)
0928 21:20:04.672505 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.672508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.673521 (+ 1013us) server_negotiation.cc:407] Connection header received
0928 21:20:04.675164 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.675168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.675218 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.675288 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.676622 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.677118 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.678388 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.678600 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.678685 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.681722 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.681725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.681728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.681797 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.683509 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.684139 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.684143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.684144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.684238 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.684709 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.684715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.684717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.684902 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.685002 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.685681 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.685791 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"spinlock_wait_cycles":3456}
I20260928 21:20:04.686807 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.672719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:40621)
0928 21:20:04.673347 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.673397 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.674916 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.675510 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.675519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.675914 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:04.676455 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.676470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.677255 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.677258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.678256 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.678268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.678761 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.678765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.678942 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.679580 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:04.679601 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:04.681467 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:04.683655 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.683663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.683685 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.684016 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.684352 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:04.684355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:04.684357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:04.684579 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:04.685023 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:04.685030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:04.685356 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.685526 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.685877 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":169,"threads_started":1}
I20260928 21:20:04.688184 21596 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34467
I20260928 21:20:04.688414 21596 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:04.688917 21596 heartbeater.cc:511] Master 127.5.18.254:34467 requested a full tablet report, sending...
I20260928 21:20:04.689930 21138 ts_manager.cc:194] Registered new tserver with Master: 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807)
I20260928 21:20:04.690600 21138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:40621
I20260928 21:20:04.697721 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:04.701858 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.698372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.5.18.254:34467)
0928 21:20:04.698491 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.698494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.698677 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:20:04.698803 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.698805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.698935 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.699032 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:04.699323 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.699873 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.700731 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.700946 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.701030 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.701295 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:04.701438 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 21:20:04.701745 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46830: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 21:20:04.701929 21328 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46830: token expired
W20260928 21:20:04.702391 21063 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.18.254:34467)
I20260928 21:20:04.709944 21328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.702824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.5.18.254:34467)
0928 21:20:04.702932 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.702936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.702949 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:20:04.703049 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.703053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.703095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.703157 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:04.703861 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.704366 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.705231 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.705444 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.705559 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.706963 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:04.706966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:04.706968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:04.706994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:04.708379 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.708784 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.708786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.708787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.708852 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:04.709078 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:04.709081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:04.709082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:04.709276 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:04.709388 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.709711 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.709819 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:20:04.712646 21138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:04.716585 21138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:04.724623 21260 tablet_service.cc:1467] Processing CreateTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 (DEFAULT_TABLE table=test-table [id=0c1d868b35494631b2f9dbc76e4a02e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:04.725322 21260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b644514c1e847f9b6513f4e4560e7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:04.728111 21218 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap starting.
I20260928 21:20:04.728902 21218 tablet_bootstrap.cc:654] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:04.729720 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.724029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.0.0.1:48318)
0928 21:20:04.724316 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.724328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.724411 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.725172 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.725174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.725204 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:04.725342 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.725348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.726704 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.726707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.727983 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.727994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.729179 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.729183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.729299 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.729359 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.729450 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.729541 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":4,"spinlock_wait_cycles":6016,"thread_start_us":127,"threads_started":1}
I20260928 21:20:04.729893 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.723655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39807 (local address 127.0.0.1:41240)
0928 21:20:04.723821 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.723835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.723907 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.725169 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.725172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.725200 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:04.725356 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.725363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.727030 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.727034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.728110 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.728118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.729425 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.729428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.729546 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.729603 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.729676 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.729760 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":40,"spinlock_wait_cycles":1792}
I20260928 21:20:04.730441 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.724079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.5.18.195:39807)
0928 21:20:04.724613 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.724623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.724647 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:20:04.724728 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.724734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.724992 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.725155 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:04.725491 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.726913 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.728323 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.729295 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.729420 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.729559 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.729711 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.729808 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":133,"threads_started":1}
I20260928 21:20:04.730526 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.724474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.5.18.194:37329)
0928 21:20:04.724805 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.724815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.724834 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:20:04.724894 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.724898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.725034 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.725527 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:04.725705 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.726582 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.728139 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.729009 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.729157 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.729320 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.730264 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.730377 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":133,"threads_started":1}
I20260928 21:20:04.730647 21218 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: No bootstrap required, opened a new log
I20260928 21:20:04.730744 21218 ts_tablet_manager.cc:1397] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:20:04.731215 21218 raft_consensus.cc:348] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.731344 21218 raft_consensus.cc:374] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:04.731388 21218 raft_consensus.cc:729] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a11d28429240fdb8fec9a001aa3720, State: Initialized, Role: FOLLOWER
I20260928 21:20:04.731525 21218 consensus_queue.cc:260] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.731913 21218 ts_tablet_manager.cc:1428] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:20:04.733672 21398 tablet_service.cc:1467] Processing CreateTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 (DEFAULT_TABLE table=test-table [id=0c1d868b35494631b2f9dbc76e4a02e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:04.733994 21530 tablet_service.cc:1467] Processing CreateTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 (DEFAULT_TABLE table=test-table [id=0c1d868b35494631b2f9dbc76e4a02e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:04.734972 21398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b644514c1e847f9b6513f4e4560e7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:04.734978 21530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b644514c1e847f9b6513f4e4560e7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:04.742707 21609 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Bootstrap starting.
I20260928 21:20:04.743929 21610 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Bootstrap starting.
I20260928 21:20:04.745187 21609 tablet_bootstrap.cc:654] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:04.745818 21610 tablet_bootstrap.cc:654] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:04.746027 21609 log.cc:824] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:04.746925 21610 log.cc:824] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:04.748178 21609 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: No bootstrap required, opened a new log
I20260928 21:20:04.748368 21609 ts_tablet_manager.cc:1397] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:20:04.748778 21610 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: No bootstrap required, opened a new log
I20260928 21:20:04.748962 21610 ts_tablet_manager.cc:1397] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:20:04.751494 21609 raft_consensus.cc:348] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.751745 21609 raft_consensus.cc:374] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:04.751780 21609 raft_consensus.cc:729] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7af4d109c6c4c68936a2ca5a6257d56, State: Initialized, Role: FOLLOWER
I20260928 21:20:04.751849 21610 raft_consensus.cc:348] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.752107 21610 raft_consensus.cc:374] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:04.752168 21610 raft_consensus.cc:729] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11051de5271e44df8a1c2e8a57c68ee7, State: Initialized, Role: FOLLOWER
I20260928 21:20:04.752206 21609 consensus_queue.cc:260] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.752545 21610 consensus_queue.cc:260] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.752871 21464 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
I20260928 21:20:04.753213 21609 ts_tablet_manager.cc:1428] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:20:04.753407 21596 heartbeater.cc:503] Master 127.5.18.254:34467 was elected leader, sending a full tablet report...
I20260928 21:20:04.753453 21610 ts_tablet_manager.cc:1428] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260928 21:20:04.791632 21327 tablet.cc:2369] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:04.799853 21614 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:04.800094 21614 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.801199 21614 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329), 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:20:04.805362 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.801619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46705 (local address 127.5.18.194:37329)
0928 21:20:04.801734 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.801739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.801754 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:20:04.801812 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.801815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.801865 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.801958 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:04.802250 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.802904 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.804199 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.804804 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.804910 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.804986 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.805163 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.805236 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:20:04.805351 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.801454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.5.18.195:46705)
0928 21:20:04.801589 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.801604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.801704 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.801987 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.801989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.802010 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:04.802147 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.802153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.802998 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.803001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.804083 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.804092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.804928 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.804932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.805034 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.805059 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.805136 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.805209 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":1,"spinlock_wait_cycles":26880}
I20260928 21:20:04.805501 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.801454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.5.18.195:52073)
0928 21:20:04.801936 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.801949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:04.802076 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:04.802339 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:04.802342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:04.802357 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:04.802463 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.802469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.803301 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.803304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.804297 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:04.804308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.805125 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.805129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:04.805221 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.805240 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:20:04.805293 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 21:20:04.805342 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":157,"spinlock_wait_cycles":768,"thread_start_us":119,"threads_started":1}
I20260928 21:20:04.805508 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.801585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:52073 (local address 127.5.18.193:45745)
0928 21:20:04.801735 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.801739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.802027 (+ 288us) server_negotiation.cc:407] Connection header received
0928 21:20:04.802166 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.802169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.802236 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.802315 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:04.802559 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.803173 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.804399 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.805025 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.805119 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.805290 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.805322 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.805392 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:20:04.806139 21418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b644514c1e847f9b6513f4e4560e7b8" candidate_uuid: "11051de5271e44df8a1c2e8a57c68ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" is_pre_election: true
I20260928 21:20:04.806223 21279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b644514c1e847f9b6513f4e4560e7b8" candidate_uuid: "11051de5271e44df8a1c2e8a57c68ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a11d28429240fdb8fec9a001aa3720" is_pre_election: true
I20260928 21:20:04.806450 21418 raft_consensus.cc:2454] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11051de5271e44df8a1c2e8a57c68ee7 in term 0.
I20260928 21:20:04.806488 21279 raft_consensus.cc:2454] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11051de5271e44df8a1c2e8a57c68ee7 in term 0.
I20260928 21:20:04.806891 21484 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56; no voters:
I20260928 21:20:04.807155 21614 raft_consensus.cc:2781] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:04.807228 21614 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:04.807258 21614 raft_consensus.cc:3037] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:04.808331 21614 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.808661 21614 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [CANDIDATE]: Term 1 election: Requested vote from peers c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329), 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745)
I20260928 21:20:04.809008 21279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b644514c1e847f9b6513f4e4560e7b8" candidate_uuid: "11051de5271e44df8a1c2e8a57c68ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51a11d28429240fdb8fec9a001aa3720"
I20260928 21:20:04.809140 21279 raft_consensus.cc:3037] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:04.809136 21418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b644514c1e847f9b6513f4e4560e7b8" candidate_uuid: "11051de5271e44df8a1c2e8a57c68ee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
I20260928 21:20:04.809253 21418 raft_consensus.cc:3037] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:04.810245 21279 raft_consensus.cc:2454] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11051de5271e44df8a1c2e8a57c68ee7 in term 1.
I20260928 21:20:04.810261 21418 raft_consensus.cc:2454] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11051de5271e44df8a1c2e8a57c68ee7 in term 1.
I20260928 21:20:04.810539 21483 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11051de5271e44df8a1c2e8a57c68ee7, 51a11d28429240fdb8fec9a001aa3720; no voters:
I20260928 21:20:04.810700 21614 raft_consensus.cc:2781] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:04.810896 21614 raft_consensus.cc:686] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 11051de5271e44df8a1c2e8a57c68ee7, State: Running, Role: LEADER
I20260928 21:20:04.811244 21614 consensus_queue.cc:237] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:04.814289 21138 catalog_manager.cc:5495] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195). New cstate: current_term: 1 leader_uuid: "11051de5271e44df8a1c2e8a57c68ee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } health_report { overall_health: UNKNOWN } } }
I20260928 21:20:04.831367 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.828035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.5.18.195:39807)
0928 21:20:04.828143 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:04.828147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:04.828164 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:20:04.828415 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:04.828418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:04.828480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:04.828574 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:04.828758 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.829239 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.830094 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:04.830301 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:04.830375 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:04.830636 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:04.831046 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:04.831130 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:04.831205 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:20:04.831255 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:20:04.837317 21279 raft_consensus.cc:1261] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Refusing update from remote peer 11051de5271e44df8a1c2e8a57c68ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:04.837320 21418 raft_consensus.cc:1261] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Refusing update from remote peer 11051de5271e44df8a1c2e8a57c68ee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:04.837981 21614 consensus_queue.cc:1035] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:04.838176 21616 consensus_queue.cc:1035] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:04.844950 21620 mvcc.cc:205] Tried to move back new op lower bound from 7334422138205941760 to 7334422138109472768. Current Snapshot: MvccSnapshot[applied={T|T < 7334422138205941760}]
I20260928 21:20:04.853183 21137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:04.853376 21137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:04.855856 21137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Sending DeleteTablet for 3 replicas of tablet 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:04.856592 21260 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:04 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:54472
I20260928 21:20:04.856655 21398 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:04 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:48318
I20260928 21:20:04.856978 21629 tablet_replica.cc:330] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: stopping tablet replica
I20260928 21:20:04.856994 21530 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b644514c1e847f9b6513f4e4560e7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:04 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:41240
I20260928 21:20:04.857121 21629 raft_consensus.cc:2229] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:04.857168 21630 tablet_replica.cc:330] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: stopping tablet replica
I20260928 21:20:04.857256 21629 raft_consensus.cc:2258] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:04.857493 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21176
I20260928 21:20:04.857503 21630 raft_consensus.cc:2229] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:04.858217 21630 raft_consensus.cc:2258] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:04.859889 21630 ts_tablet_manager.cc:1905] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:04.861795 21631 tablet_replica.cc:330] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: stopping tablet replica
I20260928 21:20:04.862185 21631 raft_consensus.cc:2229] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:04.862665 21631 raft_consensus.cc:2258] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:04.862733 21630 ts_tablet_manager.cc:1918] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:04.862836 21630 log.cc:1196] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:04.863165 21630 ts_tablet_manager.cc:1939] T 6b644514c1e847f9b6513f4e4560e7b8 P c7af4d109c6c4c68936a2ca5a6257d56: Deleting consensus metadata
W20260928 21:20:04.863734 21122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:45745 (error 108)
I20260928 21:20:04.864106 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21331
I20260928 21:20:04.864389 21631 ts_tablet_manager.cc:1905] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:04.866592 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.866294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:45745 (local address 127.0.0.1:54484)
0928 21:20:04.866427 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.866497 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 21:20:04.869761 21122 catalog_manager.cc:4584] TS 51a11d28429240fdb8fec9a001aa3720 (127.5.18.193:45745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b644514c1e847f9b6513f4e4560e7b8: Network error: Client connection negotiation failed: client connection to 127.5.18.193:45745: connect: Connection refused (error 111)
I20260928 21:20:04.870777 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21467
I20260928 21:20:04.871228 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:04.870899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.0.0.1:48334)
0928 21:20:04.871098 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:04.871145 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 21:20:04.875726 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21105
I20260928 21:20:24.881458 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--webserver_interface=127.5.18.254
--webserver_port=40133
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:24.994400 21633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:24.994696 21633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:24.994753 21633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:24.998817 21633 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:20:24.998886 21633 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:20:24.998908 21633 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:20:24.998924 21633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:24.998950 21633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:24.998983 21633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:24.999010 21633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:24.999037 21633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:25.004485 21633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:34467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=40133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:25.005676 21633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:25.006748 21633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:25.015686 21638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.015756 21633 server_base.cc:1034] running on GCE node
W20260928 21:20:25.015686 21641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:25.015688 21639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.016566 21633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:25.017750 21633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:25.018909 21633 hybrid_clock.cc:648] HybridClock initialized: now 1790630425018887 us; error 39 us; skew 500 ppm
Sep 28 21:20:25 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:25.027556 21633 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:20:25.028548 21633 webserver.cc:460] Webserver started at http://127.5.18.254:40133/ using document root <none> and password file <none>
I20260928 21:20:25.029114 21633 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:25.029184 21633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:25.033389 21633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:25.035771 21648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.037141 21633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:20:25.037266 21633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "87c3db699cdc475fbeb4d32af0b47efb"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "9c83daacecd79b2ae36d027c6a9a277c"
server_key_iv: "95cf064e3da1d7e24e1da25ce4a903f0"
server_key_version: "encryptionkey@0"
I20260928 21:20:25.037911 21633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:25.053351 21633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:25.056619 21633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:25.056823 21633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:25.063861 21633 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:34467
I20260928 21:20:25.064154 21700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:34467 every 8 connection(s)
I20260928 21:20:25.064980 21633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:25.067333 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21633
I20260928 21:20:25.067759 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:45745
--local_ip_for_outbound_sockets=127.5.18.193
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=34503
--webserver_interface=127.5.18.193
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:20:25.069548 21701 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:20:25.071801 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap starting.
I20260928 21:20:25.093811 21701 log.cc:824] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:25.105046 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:20:25.105446 21701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Bootstrap complete.
I20260928 21:20:25.109359 21701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:25.109822 21701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Initialized, Role: FOLLOWER
I20260928 21:20:25.110342 21701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:25.110466 21701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:25.110531 21701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:25.110623 21701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:20:25.112542 21701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:25.112859 21701 leader_election.cc:302] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c3db699cdc475fbeb4d32af0b47efb; no voters:
I20260928 21:20:25.113169 21701 leader_election.cc:288] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:20:25.113271 21705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:20:25.113462 21705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [term 3 LEADER]: Becoming Leader. State: Replica: 87c3db699cdc475fbeb4d32af0b47efb, State: Running, Role: LEADER
I20260928 21:20:25.113798 21705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } }
I20260928 21:20:25.114444 21701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:25.114740 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c3db699cdc475fbeb4d32af0b47efb. Latest consensus state: current_term: 3 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:20:25.114853 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:25.115443 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "87c3db699cdc475fbeb4d32af0b47efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c3db699cdc475fbeb4d32af0b47efb" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34467 } } }
I20260928 21:20:25.115578 21709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:25.115614 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:25.120751 21709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:25.121547 21709 catalog_manager.cc:1259] Loaded cluster ID: 83abcb060f4b412cad0caf20cdb4be4e
I20260928 21:20:25.121644 21709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:25.123431 21709 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:25.124027 21709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Loaded TSK: 0
I20260928 21:20:25.124094 21709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87c3db699cdc475fbeb4d32af0b47efb: Loaded TSK: 1
F20260928 21:20:25.124680 21709 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbecd7149ad google::LogMessage::Fail() at ??:0
@ 0x7fbecd71788a google::LogMessage::SendToLog() at ??:0
@ 0x7fbecd7144de google::LogMessage::Flush() at ??:0
@ 0x7fbecd715a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbed03d818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbed03d0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbed80797bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbed805399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbed8054178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbed8095df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbed80b482b std::function<>::operator()() at ??:0
@ 0x7fbed8053b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbed8054f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbed8051579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbed8095640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbecf9663d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbecf966c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbecf9683f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbecf957512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbed0f9b6db start_thread at ??:0
@ 0x7fbecc21261f clone at ??:0
W20260928 21:20:25.207348 21703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:25.207590 21703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:25.207633 21703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:25.211935 21703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:25.211988 21703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:25.212049 21703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:25.217859 21703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:45745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=34503
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:25.218957 21703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:25.219946 21703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:25.229691 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:25.229691 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.229880 21703 server_base.cc:1034] running on GCE node
W20260928 21:20:25.230075 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.230473 21703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:25.231122 21703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:25.232298 21703 hybrid_clock.cc:648] HybridClock initialized: now 1790630425232271 us; error 42 us; skew 500 ppm
Sep 28 21:20:25 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:25.235338 21703 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:20:25.236497 21703 webserver.cc:460] Webserver started at http://127.5.18.193:34503/ using document root <none> and password file <none>
I20260928 21:20:25.237237 21703 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:25.237334 21703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:25.242700 21703 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:20:25.245460 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.247028 21703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:20:25.247160 21703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "51a11d28429240fdb8fec9a001aa3720"
format_stamp: "Formatted at 2026-09-28 21:19:42 on dist-test-slave-rc9f"
server_key: "2ec3aa5638c8d289f82219c38c4bb041"
server_key_iv: "134e86a9a63879665dfe0192076601ed"
server_key_version: "encryptionkey@0"
I20260928 21:20:25.247886 21703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:25.282933 21703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:25.286850 21703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:25.287057 21703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:25.287819 21703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:25.289136 21765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:20:25.293316 21703 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:20:25.293396 21703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 21:20:25.293463 21703 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:20:25.295725 21703 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:20:25.295786 21703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:20:25.296069 21765 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap starting.
I20260928 21:20:25.306403 21703 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:45745
I20260928 21:20:25.306680 21872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:45745 every 8 connection(s)
I20260928 21:20:25.307727 21703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:25.315598 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21703
I20260928 21:20:25.315608 21765 log.cc:824] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:25.316087 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:37329
--local_ip_for_outbound_sockets=127.5.18.194
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=36589
--webserver_interface=127.5.18.194
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:20:25.323182 21765 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:20:25.323532 21765 tablet_bootstrap.cc:492] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Bootstrap complete.
I20260928 21:20:25.324445 21765 ts_tablet_manager.cc:1397] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.012s
I20260928 21:20:25.327071 21765 raft_consensus.cc:348] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:25.327483 21765 raft_consensus.cc:729] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a11d28429240fdb8fec9a001aa3720, State: Initialized, Role: FOLLOWER
I20260928 21:20:25.327958 21765 consensus_queue.cc:260] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:25.328889 21765 ts_tablet_manager.cc:1428] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260928 21:20:25.387338 21875 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:20:25.310138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:53749)
0928 21:20:25.310752 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.310821 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:25.311975 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:25.386173 (+ 74198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":153,"threads_started":1}
W20260928 21:20:25.387868 21873 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:34467 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:34467: Client connection negotiation failed: client connection to 127.5.18.254:34467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:20:25.388652 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.388355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:43393)
0928 21:20:25.388492 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.388579 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:20:25.389310 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.389100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:50609)
0928 21:20:25.389201 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.389252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:20:25.389482 21873 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:20:25.447710 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:25.448014 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:25.448061 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:25.452251 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:25.452332 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:25.452394 21877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:25.457947 21877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:37329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=36589
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:25.459019 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:25.460037 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:25.469502 21883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:25.469503 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.469604 21877 server_base.cc:1034] running on GCE node
W20260928 21:20:25.469532 21884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.470198 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:25.470850 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:25.472016 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790630425471999 us; error 34 us; skew 500 ppm
Sep 28 21:20:25 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:25.474743 21877 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:20:25.475607 21877 webserver.cc:460] Webserver started at http://127.5.18.194:36589/ using document root <none> and password file <none>
I20260928 21:20:25.476151 21877 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:25.476212 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:25.480185 21877 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:20:25.482331 21893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.483672 21877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:25.483774 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c7af4d109c6c4c68936a2ca5a6257d56"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "5bb8cb671b69f1fa77ce39f9ef3811bf"
server_key_iv: "d45cb1648dd89d5b529d9da91129c2ad"
server_key_version: "encryptionkey@0"
I20260928 21:20:25.484349 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:25.505940 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:25.508925 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:25.509115 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:25.509739 21877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:25.510596 21877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:25.510643 21877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.510689 21877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:25.510715 21877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.519089 21877 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37329
I20260928 21:20:25.519124 22006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37329 every 8 connection(s)
I20260928 21:20:25.520085 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:25.523288 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 21877
I20260928 21:20:25.523664 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:39807
--local_ip_for_outbound_sockets=127.5.18.195
--tserver_master_addrs=127.5.18.254:34467
--webserver_port=33579
--webserver_interface=127.5.18.195
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:20:25.524050 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.522187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:36303)
0928 21:20:25.522685 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.522805 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
W20260928 21:20:25.524501 22007 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:34467 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:34467: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
I20260928 21:20:25.525159 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.524907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:59287)
0928 21:20:25.525031 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.525092 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:20:25.525765 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.525543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:38065)
0928 21:20:25.525662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.525710 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 21:20:25.525929 22007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:20:25.637203 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:25.637493 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:25.637537 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:25.641799 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:25.641877 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:25.641947 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:25.647523 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:39807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=33579
--tserver_master_addrs=127.5.18.254:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:25.648672 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:25.649832 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:25.659956 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:25.659935 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:25.659940 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:25.660260 22010 server_base.cc:1034] running on GCE node
I20260928 21:20:25.660776 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:25.661454 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:25.662613 22010 hybrid_clock.cc:648] HybridClock initialized: now 1790630425662597 us; error 38 us; skew 500 ppm
Sep 28 21:20:25 dist-test-slave-rc9f krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:25.665383 22010 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:20:25.666405 22010 webserver.cc:460] Webserver started at http://127.5.18.195:33579/ using document root <none> and password file <none>
I20260928 21:20:25.666970 22010 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:25.667038 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:25.671052 22010 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 21:20:25.673305 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.674659 22010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:20:25.674783 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "11051de5271e44df8a1c2e8a57c68ee7"
format_stamp: "Formatted at 2026-09-28 21:19:43 on dist-test-slave-rc9f"
server_key: "7e0b8206f137c03d4dacc652d84f0077"
server_key_iv: "a0a0046b9ba8c17da9ffd0e5421a591f"
server_key_version: "encryptionkey@0"
I20260928 21:20:25.675360 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:25.693284 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:25.696702 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:25.696944 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:25.697604 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:25.698679 22032 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 21:20:25.702561 22032 ts_tablet_manager.cc:1852] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 21:20:25.702970 22032 ts_tablet_manager.cc:1905] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:25.707181 22032 ts_tablet_manager.cc:1918] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:25.707300 22032 log.cc:1196] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:25.707623 22032 ts_tablet_manager.cc:1939] T 6b644514c1e847f9b6513f4e4560e7b8 P 11051de5271e44df8a1c2e8a57c68ee7: Deleting consensus metadata
I20260928 21:20:25.707917 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 21:20:25.707983 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260928 21:20:25.708065 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:25.708091 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:25.716686 22010 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:39807
I20260928 21:20:25.716763 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:39807 every 8 connection(s)
I20260928 21:20:25.717644 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:25.720583 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22010
I20260928 21:20:25.721289 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.719798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:58741)
0928 21:20:25.720271 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.720415 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
W20260928 21:20:25.721722 22140 heartbeater.cc:644] Failed to heartbeat to 127.5.18.254:34467 (0 consecutive failures): Network error: Failed to ping master at 127.5.18.254:34467: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
I20260928 21:20:25.722260 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.722063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:41209)
0928 21:20:25.722167 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.722210 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 21:20:25.722782 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:25.722603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:57785)
0928 21:20:25.722694 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:25.722737 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:20:25.722920 22140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:20:26.390580 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:26.390145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:41567)
0928 21:20:26.390397 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:26.390464 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260928 21:20:26.526803 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:26.526423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:51357)
0928 21:20:26.526653 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:26.526706 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260928 21:20:26.723827 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:26.723424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:57031)
0928 21:20:26.723677 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:26.723730 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260928 21:20:27.081785 22147 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:27.082001 22147 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:27.083074 22147 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
Sep 28 21:20:27 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.194@KRBTEST.COM
Sep 28 21:20:27 dist-test-slave-rc9f krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630425, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.195@KRBTEST.COM
I20260928 21:20:27.098737 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.083345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37329 (local address 127.5.18.193:52543)
0928 21:20:27.083796 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:27.083810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:27.083923 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:27.085532 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:27.085535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:27.085938 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:27.086712 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:27.086734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.088684 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.088687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:27.089895 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:27.089908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.090896 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:27.091157 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:27.091926 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:27.091955 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:27.093811 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:27.096033 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:27.096055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:27.096428 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:27.096852 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:27.096857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:27.097038 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:27.097686 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:27.097693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:27.097969 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 21:20:27.098134 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 21:20:27.098521 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":318,"spinlock_wait_cycles":22912,"thread_start_us":79,"threads_started":1}
I20260928 21:20:27.098730 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.083467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:36935 (local address 127.5.18.195:39807)
0928 21:20:27.083897 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:27.083912 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:27.083956 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:20:27.084680 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:27.084685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:27.084985 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:27.085263 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:27.086922 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.088490 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090014 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.090749 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090912 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:27.094237 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:27.094242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:27.094264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:27.094293 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:27.095965 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096559 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:27.096566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:27.096572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:27.096742 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:27.097111 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:27.097114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:27.097116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:27.097553 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:27.097708 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:27.098276 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 21:20:27.098530 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":61,"threads_started":1}
I20260928 21:20:27.098742 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.083609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:52543 (local address 127.5.18.194:37329)
0928 21:20:27.083980 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:27.083991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:27.084020 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:20:27.084636 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:27.084642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:27.084984 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:27.085264 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:27.086922 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.088490 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090050 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.090749 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090912 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:27.094084 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:27.094092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:27.094110 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:27.094147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:27.095907 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096563 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:27.096568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:27.096572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:27.096742 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:27.097141 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:27.097144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:27.097145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:27.097554 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:27.097709 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:27.098281 (+ 572us) server_negotiation.cc:299] Negotiation successful
0928 21:20:27.098539 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":54,"threads_started":1}
I20260928 21:20:27.098775 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.083345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39807 (local address 127.5.18.193:36935)
0928 21:20:27.083727 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:27.083745 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:27.083865 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:27.085343 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:27.085349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:27.085956 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:27.086712 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:27.086735 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.088678 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.088682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:27.089895 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:27.089908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:27.090896 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:27.090900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:27.091156 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:27.091926 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:27.091962 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:27.094060 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:27.096087 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:27.096091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:27.096414 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:27.096841 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:27.096845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:27.096846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:27.097009 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:27.097688 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:27.097694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:27.097968 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 21:20:27.098144 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 21:20:27.098596 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":519,"spinlock_wait_cycles":43520,"thread_start_us":83,"threads_started":1}
W20260928 21:20:27.101258 21760 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
W20260928 21:20:27.101420 21759 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:27.101511 21759 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a11d28429240fdb8fec9a001aa3720; no voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56
I20260928 21:20:27.101786 22147 raft_consensus.cc:2726] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:20:27.391659 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.391358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:52593)
0928 21:20:27.391494 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:27.391559 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:20:27.527843 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.527452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:49163)
0928 21:20:27.527688 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:27.527740 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260928 21:20:27.725051 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:27.724584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:34227)
0928 21:20:27.724891 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:27.724951 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260928 21:20:28.393020 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:28.392524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:52371)
0928 21:20:28.392844 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:28.392915 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
I20260928 21:20:28.529052 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:28.528638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:49755)
0928 21:20:28.528890 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:28.528955 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260928 21:20:28.726325 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:28.725861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:60631)
0928 21:20:28.726158 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:28.726225 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260928 21:20:29.165923 22157 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:29.166059 22157 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:29.166453 22157 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
W20260928 21:20:29.167178 21760 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
W20260928 21:20:29.167279 21759 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:29.167368 21759 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a11d28429240fdb8fec9a001aa3720; no voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56
I20260928 21:20:29.167498 22157 raft_consensus.cc:2726] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:20:29.394349 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:29.393849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:33221)
0928 21:20:29.394130 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:29.394197 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260928 21:20:29.530220 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:29.529812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:45425)
0928 21:20:29.530035 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:29.530096 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260928 21:20:29.727411 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:29.726996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:49505)
0928 21:20:29.727255 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:29.727316 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260928 21:20:30.395643 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:30.395217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:54859)
0928 21:20:30.395487 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:30.395544 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260928 21:20:30.531490 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:30.531050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:60173)
0928 21:20:30.531319 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:30.531389 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260928 21:20:30.728652 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:30.728209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:44437)
0928 21:20:30.728481 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:30.728546 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260928 21:20:31.392771 22164 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:31.392902 22164 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:31.393262 22164 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
W20260928 21:20:31.394044 21759 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
W20260928 21:20:31.394164 21760 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:31.394225 21760 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a11d28429240fdb8fec9a001aa3720; no voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56
I20260928 21:20:31.394361 22164 raft_consensus.cc:2726] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:20:31.396644 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:31.396294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:44491)
0928 21:20:31.396500 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:31.396556 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260928 21:20:31.532811 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:31.532355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:45249)
0928 21:20:31.532640 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:31.532702 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260928 21:20:31.729879 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:31.729453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:35133)
0928 21:20:31.729714 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:31.729784 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":113,"threads_started":1}
I20260928 21:20:32.397933 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:32.397409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:60979)
0928 21:20:32.397766 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:32.397832 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":91,"threads_started":1}
I20260928 21:20:32.534005 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:32.533519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:44515)
0928 21:20:32.533840 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:32.533906 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260928 21:20:32.731055 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:32.730571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:34393)
0928 21:20:32.730897 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:32.730956 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":76,"threads_started":1}
I20260928 21:20:33.399204 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:33.398755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:59407)
0928 21:20:33.399031 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:33.399092 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260928 21:20:33.515822 22172 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:33.515981 22172 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:33.516470 22172 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
W20260928 21:20:33.517167 21759 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
W20260928 21:20:33.517266 21760 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:33.517364 21760 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a11d28429240fdb8fec9a001aa3720; no voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56
I20260928 21:20:33.517501 22172 raft_consensus.cc:2726] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:20:33.535243 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:33.534810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:37719)
0928 21:20:33.535081 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:33.535144 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260928 21:20:33.732337 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:33.731891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:50755)
0928 21:20:33.732174 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:33.732239 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260928 21:20:34.400410 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:34.399990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:36179)
0928 21:20:34.400250 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:34.400312 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260928 21:20:34.536412 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:34.535972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:56365)
0928 21:20:34.536245 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:34.536308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260928 21:20:34.733824 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:34.733273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:48577)
0928 21:20:34.733638 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:34.733717 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
I20260928 21:20:35.401772 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:35.401313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:35801)
0928 21:20:35.401607 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:35.401673 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260928 21:20:35.537676 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:35.537202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:49961)
0928 21:20:35.537464 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:35.537522 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260928 21:20:35.735025 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:35.734595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:47525)
0928 21:20:35.734866 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:35.734930 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20260928 21:20:36.402933 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:36.402479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:59569)
0928 21:20:36.402771 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:36.402832 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260928 21:20:36.538980 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:36.538525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:54721)
0928 21:20:36.538816 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:36.538880 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260928 21:20:36.736236 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:36.735800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:52395)
0928 21:20:36.736055 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:36.736118 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260928 21:20:37.404166 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:37.403700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:35493)
0928 21:20:37.403983 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:37.404057 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
I20260928 21:20:37.540133 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:37.539690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:55893)
0928 21:20:37.539974 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:37.540036 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260928 21:20:37.737344 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:37.736949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:35569)
0928 21:20:37.737193 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:37.737250 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260928 21:20:38.372279 22187 raft_consensus.cc:482] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:38.372421 22187 raft_consensus.cc:504] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11051de5271e44df8a1c2e8a57c68ee7" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39807 } } peers { permanent_uuid: "c7af4d109c6c4c68936a2ca5a6257d56" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37329 } } peers { permanent_uuid: "51a11d28429240fdb8fec9a001aa3720" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 45745 } }
I20260928 21:20:38.372848 22187 leader_election.cc:288] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807), c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329)
W20260928 21:20:38.373692 21759 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11051de5271e44df8a1c2e8a57c68ee7 (127.5.18.195:39807): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
W20260928 21:20:38.373884 21760 leader_election.cc:341] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c7af4d109c6c4c68936a2ca5a6257d56 (127.5.18.194:37329): Not found: Tablet not found: 6b644514c1e847f9b6513f4e4560e7b8
I20260928 21:20:38.373970 21760 leader_election.cc:302] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 51a11d28429240fdb8fec9a001aa3720; no voters: 11051de5271e44df8a1c2e8a57c68ee7, c7af4d109c6c4c68936a2ca5a6257d56
I20260928 21:20:38.374126 22187 raft_consensus.cc:2726] T 6b644514c1e847f9b6513f4e4560e7b8 P 51a11d28429240fdb8fec9a001aa3720 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:20:38.405335 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:38.404916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:53925)
0928 21:20:38.405180 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:38.405240 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260928 21:20:38.541455 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:38.540990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:55121)
0928 21:20:38.541294 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:38.541356 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260928 21:20:38.738543 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:38.738126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:43229)
0928 21:20:38.738382 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:38.738444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260928 21:20:39.406574 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:39.406145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:32933)
0928 21:20:39.406412 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:39.406475 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260928 21:20:39.542719 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:39.542283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:44837)
0928 21:20:39.542554 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:39.542615 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260928 21:20:39.739651 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:39.739238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.195:38303)
0928 21:20:39.739499 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:39.739557 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260928 21:20:40.407908 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:40.407514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.193:44241)
0928 21:20:40.407758 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:40.407813 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260928 21:20:40.544123 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:40.543698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34467 (local address 127.5.18.194:46059)
0928 21:20:40.543962 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:40.544025 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.254:34467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260928 21:20:40.722404 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21703
I20260928 21:20:40.727625 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 21877
I20260928 21:20:40.731912 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22010
2026-09-28T21:20:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59331 ms)
[----------] 30 tests from SecurityITest (234574 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:40 dist-test-slave-rc9f krb5kdc[22198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:40 dist-test-slave-rc9f krb5kdc[22198](info): set up 2 sockets
Sep 28 21:20:40 dist-test-slave-rc9f krb5kdc[22198](info): commencing operation
krb5kdc: starting...
W20260928 21:20:41.689415 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:41 dist-test-slave-rc9f krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:20:41.804960 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:35121
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:35121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:41.912868 22211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:41.913147 22211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:41.913203 22211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:41.917270 22211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:41.917335 22211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:41.917367 22211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:41.917390 22211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:41.917419 22211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:41.922762 22211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:35121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:35121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:41.923832 22211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:41.924851 22211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:20:41.933394 22211 server_base.cc:1034] running on GCE node
W20260928 21:20:41.933305 22216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:41.933314 22219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:41.933321 22217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:41.934240 22211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:41.935349 22211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:41.936525 22211 hybrid_clock.cc:648] HybridClock initialized: now 1790630441936498 us; error 36 us; skew 500 ppm
Sep 28 21:20:41 dist-test-slave-rc9f krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:41.939203 22211 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:20:41.940080 22211 webserver.cc:460] Webserver started at http://127.5.18.254:42823/ using document root <none> and password file <none>
I20260928 21:20:41.940631 22211 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:41.940697 22211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:41.940878 22211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:41.942945 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "3333a9da191044daa7de5301565c507d"
format_stamp: "Formatted at 2026-09-28 21:20:41 on dist-test-slave-rc9f"
server_key: "cc2601c07e9fd5eca803bc7c13836613"
server_key_iv: "f6ee459d6a8c115d46e2f025dddd7dbf"
server_key_version: "encryptionkey@0"
I20260928 21:20:41.943488 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "3333a9da191044daa7de5301565c507d"
format_stamp: "Formatted at 2026-09-28 21:20:41 on dist-test-slave-rc9f"
server_key: "cc2601c07e9fd5eca803bc7c13836613"
server_key_iv: "f6ee459d6a8c115d46e2f025dddd7dbf"
server_key_version: "encryptionkey@0"
I20260928 21:20:41.947283 22211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:20:41.949673 22226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:41.950896 22211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:41.951013 22211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "3333a9da191044daa7de5301565c507d"
format_stamp: "Formatted at 2026-09-28 21:20:41 on dist-test-slave-rc9f"
server_key: "cc2601c07e9fd5eca803bc7c13836613"
server_key_iv: "f6ee459d6a8c115d46e2f025dddd7dbf"
server_key_version: "encryptionkey@0"
I20260928 21:20:41.951121 22211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:41.972867 22211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:41.976044 22211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:41.976228 22211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:41.983006 22211 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:35121
I20260928 21:20:41.983017 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:35121 every 8 connection(s)
I20260928 21:20:41.983943 22211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:41.986666 22279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:41.991000 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22211
I20260928 21:20:41.991122 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:41.991379 5195 external_mini_cluster.cc:1428] Setting key e60c2bea54b5ffc68229965639a94c39
I20260928 21:20:41.992369 22279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: Bootstrap starting.
I20260928 21:20:41.995007 22279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:41.995833 22279 log.cc:824] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:41.997893 22279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: No bootstrap required, opened a new log
Sep 28 21:20:41 dist-test-slave-rc9f krb5kdc[22198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:42.001914 22279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } }
I20260928 21:20:42.002173 22279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:42.002242 22279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3333a9da191044daa7de5301565c507d, State: Initialized, Role: FOLLOWER
I20260928 21:20:42.002723 22279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } }
I20260928 21:20:42.002852 22279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:42.002911 22279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:42.003000 22279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:42.004033 22279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } }
I20260928 21:20:42.004357 22279 leader_election.cc:302] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3333a9da191044daa7de5301565c507d; no voters:
I20260928 21:20:42.004643 22279 leader_election.cc:288] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:42.004740 22284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:42.004947 22284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [term 1 LEADER]: Becoming Leader. State: Replica: 3333a9da191044daa7de5301565c507d, State: Running, Role: LEADER
I20260928 21:20:42.005317 22284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } }
I20260928 21:20:42.006074 22279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:42.006789 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:41.992812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.5.18.254:35121)
0928 21:20:41.993235 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:41.993253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:41.993299 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:20:41.994149 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:41.994172 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:41.994605 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:41.994960 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:41.995621 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:41.996921 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:41.997870 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:41.998601 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:41.998766 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.001486 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:42.001491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:42.001508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:42.001538 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:42.003501 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.003989 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.003994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.003999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.004119 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.004404 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.004408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.004410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.004905 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:42.005089 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.005524 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.005894 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260928 21:20:42.007087 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3333a9da191044daa7de5301565c507d. Latest consensus state: current_term: 1 leader_uuid: "3333a9da191044daa7de5301565c507d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } } }
I20260928 21:20:42.007507 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:42.007408 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3333a9da191044daa7de5301565c507d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3333a9da191044daa7de5301565c507d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 35121 } } }
I20260928 21:20:42.007827 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d [sys.catalog]: This master's current role is: LEADER
W20260928 21:20:42.011045 22299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:20:42.011132 22299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:20:42.011217 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:42.011986 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:42.017130 22295 catalog_manager.cc:1347] Generated new cluster ID: 299f91ef98d44fc78d2879701b389070
I20260928 21:20:42.017195 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:42.024816 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:42.025905 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:42.034518 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: Generated new TSK 0
I20260928 21:20:42.035239 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:20:42.064101 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:42.194072 22305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:42.194365 22305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:42.194411 22305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:42.198382 22305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:42.198455 22305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:42.198515 22305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:42.203814 22305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:42.204867 22305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:42.205899 22305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:42.215420 22310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:42.215456 22313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:42.215518 22311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:42.215788 22305 server_base.cc:1034] running on GCE node
I20260928 21:20:42.216193 22305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:42.216830 22305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:42.218001 22305 hybrid_clock.cc:648] HybridClock initialized: now 1790630442217970 us; error 45 us; skew 500 ppm
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:42.220722 22305 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:20:42.221594 22305 webserver.cc:460] Webserver started at http://127.5.18.193:44185/ using document root <none> and password file <none>
I20260928 21:20:42.222134 22305 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:42.222194 22305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:42.222357 22305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:42.224349 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "2b0c8d45cb984af5a09ded8e0e834463"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "009f7bf3179faa9861dcbae83852374d"
server_key_iv: "4583415d708f44a4fe6e67fa20e5d3c7"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.224855 22305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0c8d45cb984af5a09ded8e0e834463"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "009f7bf3179faa9861dcbae83852374d"
server_key_iv: "4583415d708f44a4fe6e67fa20e5d3c7"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.228534 22305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:20:42.231083 22320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.232414 22305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:20:42.232515 22305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "2b0c8d45cb984af5a09ded8e0e834463"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "009f7bf3179faa9861dcbae83852374d"
server_key_iv: "4583415d708f44a4fe6e67fa20e5d3c7"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.232604 22305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:42.265400 22305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:42.268743 22305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:42.268914 22305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:42.269519 22305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:42.270398 22305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:42.270455 22305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.270509 22305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:42.270536 22305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.279147 22305 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:39299
I20260928 21:20:42.279174 22433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:39299 every 8 connection(s)
I20260928 21:20:42.280032 22305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:42.281303 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22305
I20260928 21:20:42.281435 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:42.281833 5195 external_mini_cluster.cc:1428] Setting key 2ab551d93db580b24bf690c212781d67
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:20:42.294298 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.281949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:35807 (local address 127.5.18.254:35121)
0928 21:20:42.282081 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.282084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.283157 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 21:20:42.284269 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.284273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.284326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.284409 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:42.285977 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.286760 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.287929 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.288286 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.288402 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.290940 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:42.290944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:42.290946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:42.290977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:42.292205 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.292748 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.292750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.292752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.292819 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.293112 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.293114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.293115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.293291 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:42.293372 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.293965 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.294109 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:20:42.295287 22436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.282461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:35121 (local address 127.5.18.193:35807)
0928 21:20:42.282996 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.283062 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.284046 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.284592 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.284602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.285104 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:42.285822 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.285843 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.286892 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.286894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.287806 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.287815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.288424 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.288427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.288566 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.289181 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:42.289199 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:42.290720 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:42.292315 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.292320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.292335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.292598 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.292887 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.292890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.292892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.293043 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.293369 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:42.293375 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:42.293721 (+ 346us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.293902 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.294287 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":122,"threads_started":1}
I20260928 21:20:42.296583 22434 heartbeater.cc:347] Connected to a master server at 127.5.18.254:35121
I20260928 21:20:42.296834 22434 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:42.297308 22434 heartbeater.cc:511] Master 127.5.18.254:35121 requested a full tablet report, sending...
I20260928 21:20:42.299085 22243 ts_manager.cc:194] Registered new tserver with Master: 2b0c8d45cb984af5a09ded8e0e834463 (127.5.18.193:39299)
I20260928 21:20:42.300634 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:35807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:20:42.309322 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:42.420044 22439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:42.420320 22439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:42.420364 22439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:42.424495 22439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:42.424563 22439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:42.424633 22439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:42.430104 22439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:42.431175 22439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:42.432139 22439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:42.441607 22444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:42.441690 22445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:42.441643 22439 server_base.cc:1034] running on GCE node
W20260928 21:20:42.441604 22447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:42.442226 22439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:42.442818 22439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:42.443984 22439 hybrid_clock.cc:648] HybridClock initialized: now 1790630442443968 us; error 29 us; skew 500 ppm
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:42.446645 22439 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:20:42.447508 22439 webserver.cc:460] Webserver started at http://127.5.18.194:34815/ using document root <none> and password file <none>
I20260928 21:20:42.448030 22439 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:42.448089 22439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:42.448283 22439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:42.450450 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "9f608f96d74a4a67abcfb99442830d2e"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "186a18b2a4b09e3cf4881e8990ce7409"
server_key_iv: "53a6d0a3a60f74d438bde6f2ccc90bdc"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.450976 22439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "9f608f96d74a4a67abcfb99442830d2e"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "186a18b2a4b09e3cf4881e8990ce7409"
server_key_iv: "53a6d0a3a60f74d438bde6f2ccc90bdc"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.454721 22439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:42.457118 22454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.458402 22439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:42.458508 22439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "9f608f96d74a4a67abcfb99442830d2e"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "186a18b2a4b09e3cf4881e8990ce7409"
server_key_iv: "53a6d0a3a60f74d438bde6f2ccc90bdc"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.458596 22439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:42.478312 22439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:42.481644 22439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:42.481804 22439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:42.482358 22439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:42.483142 22439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:42.483188 22439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.483227 22439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:42.483245 22439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.491285 22439 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:37395
I20260928 21:20:42.491313 22567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:37395 every 8 connection(s)
I20260928 21:20:42.492168 22439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:42.495559 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22439
I20260928 21:20:42.495697 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:42.496016 5195 external_mini_cluster.cc:1428] Setting key 324032988e9ab416dea234a3bae45e23
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
I20260928 21:20:42.506132 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.493896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45173 (local address 127.5.18.254:35121)
0928 21:20:42.494051 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.494056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.495006 (+ 950us) server_negotiation.cc:407] Connection header received
0928 21:20:42.496130 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.496134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.496187 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.496258 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:42.497483 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.498053 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.499466 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.499814 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.499928 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.502571 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:42.502575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:42.502578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:42.502608 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:42.504109 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.504583 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.504588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.504590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.504693 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.504981 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.504984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.504986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.505236 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:42.505337 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.505885 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.506004 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:20:42.507053 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.494304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:35121 (local address 127.5.18.194:45173)
0928 21:20:42.494792 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.494837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.495879 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.496437 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.496445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.496840 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:42.497334 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.497352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.498179 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.498183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.499339 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.499348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.499983 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.499986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.500135 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.500749 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:42.500765 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:42.502400 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:42.504221 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.504228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.504241 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.504486 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.504777 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.504778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.504780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.504903 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.505330 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:42.505335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:42.505620 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.505809 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.506175 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:20:42.508402 22568 heartbeater.cc:347] Connected to a master server at 127.5.18.254:35121
I20260928 21:20:42.508661 22568 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:42.509234 22568 heartbeater.cc:511] Master 127.5.18.254:35121 requested a full tablet report, sending...
I20260928 21:20:42.510409 22243 ts_manager.cc:194] Registered new tserver with Master: 9f608f96d74a4a67abcfb99442830d2e (127.5.18.194:37395)
I20260928 21:20:42.510962 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:45173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:20:42.524308 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:42.632347 22573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:42.632630 22573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:42.632690 22573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:42.636744 22573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:42.636828 22573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:42.636896 22573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:42.642357 22573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:35121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:42.643388 22573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:42.644371 22573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:42.653893 22581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:42.653862 22578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:42.653915 22579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:42.654256 22573 server_base.cc:1034] running on GCE node
I20260928 21:20:42.654716 22573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:42.655352 22573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:42.656503 22573 hybrid_clock.cc:648] HybridClock initialized: now 1790630442656461 us; error 52 us; skew 500 ppm
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:42.659173 22573 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:20:42.660080 22573 webserver.cc:460] Webserver started at http://127.5.18.195:40675/ using document root <none> and password file <none>
I20260928 21:20:42.660634 22573 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:42.660698 22573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:42.660890 22573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:42.663056 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "ac692ae1e2da4cbf89faca07986d6741"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "a5764718b270e3a59b4e949feaf180b8"
server_key_iv: "38cc9710a3d246a8ecc7dde9007f7ae1"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.663609 22573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "ac692ae1e2da4cbf89faca07986d6741"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "a5764718b270e3a59b4e949feaf180b8"
server_key_iv: "38cc9710a3d246a8ecc7dde9007f7ae1"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.667425 22573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:20:42.669889 22588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.671084 22573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:20:42.671196 22573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "ac692ae1e2da4cbf89faca07986d6741"
format_stamp: "Formatted at 2026-09-28 21:20:42 on dist-test-slave-rc9f"
server_key: "a5764718b270e3a59b4e949feaf180b8"
server_key_iv: "38cc9710a3d246a8ecc7dde9007f7ae1"
server_key_version: "encryptionkey@0"
I20260928 21:20:42.671312 22573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:42.681844 22573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:42.684768 22573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:42.684944 22573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:42.685503 22573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:42.686318 22573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:42.686364 22573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.686415 22573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:42.686444 22573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:42.694540 22573 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:42717
I20260928 21:20:42.694567 22701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:42717 every 8 connection(s)
I20260928 21:20:42.695394 22573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:42.700058 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22573
I20260928 21:20:42.700186 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:42.700482 5195 external_mini_cluster.cc:1428] Setting key 8f5c6d32985ac98fb164beb5c0dbaa92
Sep 28 21:20:42 dist-test-slave-rc9f krb5kdc[22198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630442, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:42.707903 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.697052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:38113 (local address 127.5.18.254:35121)
0928 21:20:42.697233 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.697238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.698090 (+ 852us) server_negotiation.cc:407] Connection header received
0928 21:20:42.698929 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.698932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.698979 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.699049 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:42.700327 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.700956 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.702344 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.702604 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.702679 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.704906 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:42.704909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:42.704911 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:42.704934 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:42.706091 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.706583 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.706585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.706586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.706670 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.706943 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.706945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.706946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.707115 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:42.707188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.707657 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.707763 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:20:42.708704 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.697460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:35121 (local address 127.5.18.195:38113)
0928 21:20:42.697942 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.697983 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.698722 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.699197 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.699203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.699548 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:42.700164 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.700186 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.701067 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.701071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.702219 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.702240 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.702732 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.702736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.702884 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.703426 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:42.703442 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:42.704766 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:42.706176 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.706181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.706194 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.706509 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.706747 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:42.706749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:42.706751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:42.706876 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:42.707199 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:42.707203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:42.707460 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.707611 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.707908 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260928 21:20:42.709750 22702 heartbeater.cc:347] Connected to a master server at 127.5.18.254:35121
I20260928 21:20:42.709970 22702 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:42.710423 22702 heartbeater.cc:511] Master 127.5.18.254:35121 requested a full tablet report, sending...
I20260928 21:20:42.711371 22243 ts_manager.cc:194] Registered new tserver with Master: ac692ae1e2da4cbf89faca07986d6741 (127.5.18.195:42717)
I20260928 21:20:42.711894 22243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:38113
I20260928 21:20:42.713966 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:42.722237 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.715732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.5.18.254:35121)
0928 21:20:42.715859 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.715863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.715998 (+ 135us) server_negotiation.cc:407] Connection header received
0928 21:20:42.716084 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.716086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.716121 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.716179 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:42.716788 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.717277 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.718155 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.718379 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.718455 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.719615 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:42.719618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:42.719620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:42.719639 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:42.720863 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.721160 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.721162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.721163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.721238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:42.721399 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:42.721400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:42.721401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:42.721657 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:42.721768 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.721989 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.722082 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:42.724918 22243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:42.727855 22243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:42.741225 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.736607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.5.18.194:37395)
0928 21:20:42.737476 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.737487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.737506 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:20:42.737596 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.737602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.737741 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.737856 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:42.738046 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738814 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.739973 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.740521 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.740614 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.740764 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.740968 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.741069 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":99,"threads_started":1}
I20260928 21:20:42.741359 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.735459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39299 (local address 127.0.0.1:60480)
0928 21:20:42.736068 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.736122 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.736270 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.737089 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.737095 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.737116 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:42.737233 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.737238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738679 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.738683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.739838 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.739849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.740961 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.740969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.741055 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.741077 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.741125 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.741189 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260928 21:20:42.741218 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.736202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:37395 (local address 127.0.0.1:49708)
0928 21:20:42.736810 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.736822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.736893 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.737853 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.737855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.737870 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:42.737949 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.737954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738931 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.738934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.739865 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.739872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.740629 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.740632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.740763 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.740806 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.740907 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.741074 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":36,"thread_start_us":86,"threads_started":1}
I20260928 21:20:42.741775 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.735596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.5.18.193:39299)
0928 21:20:42.735859 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.735877 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.736415 (+ 538us) server_negotiation.cc:407] Connection header received
0928 21:20:42.736498 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.736504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.736726 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.736875 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:42.737335 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738532 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.739970 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.740775 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.741336 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.741479 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.741525 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.741639 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260928 21:20:42.742386 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.735519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:42717 (local address 127.0.0.1:48272)
0928 21:20:42.736665 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.736679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.736792 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.737359 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.737362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.737380 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:42.737470 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.737475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738765 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.738768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.740790 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.740800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.742049 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.742053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.742117 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.742133 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.742193 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.742267 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":958,"thread_start_us":71,"threads_started":1}
I20260928 21:20:42.742444 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.736270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.5.18.195:42717)
0928 21:20:42.737014 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.737024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.737044 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:20:42.737110 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.737114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.737237 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.737362 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:42.737588 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.738631 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.740907 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.741768 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.741867 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.741994 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.742236 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.742322 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":138,"threads_started":1}
I20260928 21:20:42.744850 22502 tablet_service.cc:1467] Processing CreateTablet for tablet fa8d4ed0ee31401db569d910b5168efd (DEFAULT_TABLE table=test-table [id=8f138919dce14b25bcab39251065b662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:42.744850 22368 tablet_service.cc:1467] Processing CreateTablet for tablet fa8d4ed0ee31401db569d910b5168efd (DEFAULT_TABLE table=test-table [id=8f138919dce14b25bcab39251065b662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:42.746037 22502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8d4ed0ee31401db569d910b5168efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:42.746037 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8d4ed0ee31401db569d910b5168efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:42.747938 22636 tablet_service.cc:1467] Processing CreateTablet for tablet fa8d4ed0ee31401db569d910b5168efd (DEFAULT_TABLE table=test-table [id=8f138919dce14b25bcab39251065b662]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:42.748929 22636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8d4ed0ee31401db569d910b5168efd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:42.754248 22720 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Bootstrap starting.
I20260928 21:20:42.755451 22721 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Bootstrap starting.
I20260928 21:20:42.756011 22719 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Bootstrap starting.
I20260928 21:20:42.756829 22720 tablet_bootstrap.cc:654] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:42.757627 22721 tablet_bootstrap.cc:654] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:42.757717 22719 tablet_bootstrap.cc:654] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:42.757704 22720 log.cc:824] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:42.758634 22721 log.cc:824] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:42.758726 22719 log.cc:824] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:42.759864 22720 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: No bootstrap required, opened a new log
I20260928 21:20:42.760047 22720 ts_tablet_manager.cc:1397] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260928 21:20:42.760514 22721 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: No bootstrap required, opened a new log
I20260928 21:20:42.760696 22721 ts_tablet_manager.cc:1397] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 21:20:42.760788 22719 tablet_bootstrap.cc:492] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: No bootstrap required, opened a new log
I20260928 21:20:42.760952 22719 ts_tablet_manager.cc:1397] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 21:20:42.763211 22720 raft_consensus.cc:348] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.763437 22720 raft_consensus.cc:374] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:42.763511 22720 raft_consensus.cc:729] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f608f96d74a4a67abcfb99442830d2e, State: Initialized, Role: FOLLOWER
I20260928 21:20:42.763917 22720 consensus_queue.cc:260] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.764374 22721 raft_consensus.cc:348] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.764475 22719 raft_consensus.cc:348] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.764635 22721 raft_consensus.cc:374] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:42.764662 22719 raft_consensus.cc:374] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:42.764703 22719 raft_consensus.cc:729] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0c8d45cb984af5a09ded8e0e834463, State: Initialized, Role: FOLLOWER
I20260928 21:20:42.764748 22721 raft_consensus.cc:729] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac692ae1e2da4cbf89faca07986d6741, State: Initialized, Role: FOLLOWER
I20260928 21:20:42.764757 22720 ts_tablet_manager.cc:1428] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260928 21:20:42.765120 22568 heartbeater.cc:503] Master 127.5.18.254:35121 was elected leader, sending a full tablet report...
I20260928 21:20:42.765113 22721 consensus_queue.cc:260] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.765113 22719 consensus_queue.cc:260] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.765960 22434 heartbeater.cc:503] Master 127.5.18.254:35121 was elected leader, sending a full tablet report...
I20260928 21:20:42.766036 22719 ts_tablet_manager.cc:1428] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 21:20:42.766034 22721 ts_tablet_manager.cc:1428] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260928 21:20:42.766399 22702 heartbeater.cc:503] Master 127.5.18.254:35121 was elected leader, sending a full tablet report...
W20260928 21:20:42.781177 22435 tablet.cc:2369] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:42.934492 22725 raft_consensus.cc:482] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:42.934713 22725 raft_consensus.cc:504] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.935922 22725 leader_election.cc:288] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0c8d45cb984af5a09ded8e0e834463 (127.5.18.193:39299), ac692ae1e2da4cbf89faca07986d6741 (127.5.18.195:42717)
I20260928 21:20:42.939666 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.936457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45921 (local address 127.5.18.195:42717)
0928 21:20:42.936595 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.936599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.936700 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:20:42.936788 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.936791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.936836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.936906 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:42.937218 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.937749 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.938747 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.939138 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.939241 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.939288 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.939465 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.939534 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:20:42.939677 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.936218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39299 (local address 127.5.18.194:32937)
0928 21:20:42.936418 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.936432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.936547 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.936815 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.936817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.936831 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:42.936952 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.936958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.937757 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.937761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.938584 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.938594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.939233 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.939237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.939374 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.939391 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.939459 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.939534 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":8}
I20260928 21:20:42.939687 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.936299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:32937 (local address 127.5.18.193:39299)
0928 21:20:42.936422 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.936426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.936558 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:20:42.936647 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.936650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.936703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.936796 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:42.937041 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.937650 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.938717 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.939118 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.939217 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.939270 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.939492 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.939566 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:20:42.939710 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.936218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:42717 (local address 127.5.18.194:45921)
0928 21:20:42.936619 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:42.936631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:42.936710 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:42.936949 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:42.936952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:42.936972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:42.937096 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.937102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.937859 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.937861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.938631 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:42.938640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.939239 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.939243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:42.939353 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.939373 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:20:42.939430 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 21:20:42.939503 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":33,"thread_start_us":130,"threads_started":1}
I20260928 21:20:42.940322 22388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8d4ed0ee31401db569d910b5168efd" candidate_uuid: "9f608f96d74a4a67abcfb99442830d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0c8d45cb984af5a09ded8e0e834463" is_pre_election: true
I20260928 21:20:42.940323 22656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8d4ed0ee31401db569d910b5168efd" candidate_uuid: "9f608f96d74a4a67abcfb99442830d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac692ae1e2da4cbf89faca07986d6741" is_pre_election: true
I20260928 21:20:42.940598 22388 raft_consensus.cc:2454] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f608f96d74a4a67abcfb99442830d2e in term 0.
I20260928 21:20:42.940601 22656 raft_consensus.cc:2454] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f608f96d74a4a67abcfb99442830d2e in term 0.
I20260928 21:20:42.941039 22456 leader_election.cc:302] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0c8d45cb984af5a09ded8e0e834463, 9f608f96d74a4a67abcfb99442830d2e; no voters:
I20260928 21:20:42.941308 22725 raft_consensus.cc:2781] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:42.941392 22725 raft_consensus.cc:482] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:42.941442 22725 raft_consensus.cc:3037] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:42.942710 22725 raft_consensus.cc:504] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.943077 22725 leader_election.cc:288] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [CANDIDATE]: Term 1 election: Requested vote from peers 2b0c8d45cb984af5a09ded8e0e834463 (127.5.18.193:39299), ac692ae1e2da4cbf89faca07986d6741 (127.5.18.195:42717)
I20260928 21:20:42.943516 22388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8d4ed0ee31401db569d910b5168efd" candidate_uuid: "9f608f96d74a4a67abcfb99442830d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0c8d45cb984af5a09ded8e0e834463"
I20260928 21:20:42.943527 22656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8d4ed0ee31401db569d910b5168efd" candidate_uuid: "9f608f96d74a4a67abcfb99442830d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac692ae1e2da4cbf89faca07986d6741"
I20260928 21:20:42.943657 22656 raft_consensus.cc:3037] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:42.943675 22388 raft_consensus.cc:3037] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:42.944928 22656 raft_consensus.cc:2454] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f608f96d74a4a67abcfb99442830d2e in term 1.
I20260928 21:20:42.944933 22388 raft_consensus.cc:2454] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f608f96d74a4a67abcfb99442830d2e in term 1.
I20260928 21:20:42.945272 22456 leader_election.cc:302] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0c8d45cb984af5a09ded8e0e834463, 9f608f96d74a4a67abcfb99442830d2e; no voters:
I20260928 21:20:42.945489 22725 raft_consensus.cc:2781] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:42.945765 22725 raft_consensus.cc:686] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 1 LEADER]: Becoming Leader. State: Replica: 9f608f96d74a4a67abcfb99442830d2e, State: Running, Role: LEADER
W20260928 21:20:42.946251 22703 tablet.cc:2369] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:42.946944 22725 consensus_queue.cc:237] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } }
I20260928 21:20:42.950532 22242 catalog_manager.cc:5495] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f608f96d74a4a67abcfb99442830d2e (127.5.18.194). New cstate: current_term: 1 leader_uuid: "9f608f96d74a4a67abcfb99442830d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f608f96d74a4a67abcfb99442830d2e" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 37395 } health_report { overall_health: HEALTHY } } }
I20260928 21:20:42.978461 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:42.974934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.5.18.194:37395)
0928 21:20:42.975059 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:42.975062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:42.975076 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:20:42.975150 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:42.975152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:42.975207 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:42.975327 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:42.975533 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.976090 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.977028 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:42.977250 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:42.977329 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:42.977584 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:42.978065 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:42.978153 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:42.978241 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 21:20:42.978297 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:42.984628 22388 raft_consensus.cc:1261] T fa8d4ed0ee31401db569d910b5168efd P 2b0c8d45cb984af5a09ded8e0e834463 [term 1 FOLLOWER]: Refusing update from remote peer 9f608f96d74a4a67abcfb99442830d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:42.984645 22656 raft_consensus.cc:1261] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 1 FOLLOWER]: Refusing update from remote peer 9f608f96d74a4a67abcfb99442830d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:42.985364 22729 consensus_queue.cc:1035] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0c8d45cb984af5a09ded8e0e834463" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:42.985548 22725 consensus_queue.cc:1035] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac692ae1e2da4cbf89faca07986d6741" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:42.993764 22733 mvcc.cc:205] Tried to move back new op lower bound from 7334422294457053184 to 7334422294314131456. Current Snapshot: MvccSnapshot[applied={T|T < 7334422294457053184}]
I20260928 21:20:42.998256 22734 mvcc.cc:205] Tried to move back new op lower bound from 7334422294457053184 to 7334422294314131456. Current Snapshot: MvccSnapshot[applied={T|T < 7334422294457053184}]
I20260928 21:20:43.005040 22242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:43.005193 22242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:43.007607 22242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3333a9da191044daa7de5301565c507d: Sending DeleteTablet for 3 replicas of tablet fa8d4ed0ee31401db569d910b5168efd
I20260928 21:20:43.008384 22636 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8d4ed0ee31401db569d910b5168efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:48272
I20260928 21:20:43.008453 22368 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8d4ed0ee31401db569d910b5168efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:60480
I20260928 21:20:43.008567 22502 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8d4ed0ee31401db569d910b5168efd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:43 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:49708
I20260928 21:20:43.008940 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22305
I20260928 21:20:43.008958 22743 tablet_replica.cc:330] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: stopping tablet replica
I20260928 21:20:43.009137 22741 tablet_replica.cc:330] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: stopping tablet replica
I20260928 21:20:43.009243 22743 raft_consensus.cc:2229] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:43.009392 22741 raft_consensus.cc:2229] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:43.009691 22743 raft_consensus.cc:2258] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:43.009711 22741 raft_consensus.cc:2258] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:43.011116 22743 ts_tablet_manager.cc:1905] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:43.011283 22741 ts_tablet_manager.cc:1905] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:43.013842 22743 ts_tablet_manager.cc:1918] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:43.013964 22741 ts_tablet_manager.cc:1918] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:43.014030 22743 log.cc:1196] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/fa8d4ed0ee31401db569d910b5168efd
I20260928 21:20:43.014166 22741 log.cc:1196] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/fa8d4ed0ee31401db569d910b5168efd
I20260928 21:20:43.014343 22743 ts_tablet_manager.cc:1939] T fa8d4ed0ee31401db569d910b5168efd P 9f608f96d74a4a67abcfb99442830d2e: Deleting consensus metadata
I20260928 21:20:43.014560 22741 ts_tablet_manager.cc:1939] T fa8d4ed0ee31401db569d910b5168efd P ac692ae1e2da4cbf89faca07986d6741: Deleting consensus metadata
I20260928 21:20:43.015581 22228 catalog_manager.cc:4841] TS 9f608f96d74a4a67abcfb99442830d2e (127.5.18.194:37395): tablet fa8d4ed0ee31401db569d910b5168efd (table test-table [id=8f138919dce14b25bcab39251065b662]) successfully deleted
I20260928 21:20:43.015748 22229 catalog_manager.cc:4841] TS ac692ae1e2da4cbf89faca07986d6741 (127.5.18.195:42717): tablet fa8d4ed0ee31401db569d910b5168efd (table test-table [id=8f138919dce14b25bcab39251065b662]) successfully deleted
W20260928 21:20:43.018043 22228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:39299 (error 108)
I20260928 21:20:43.018795 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22439
I20260928 21:20:43.019172 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.018835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39299 (local address 127.0.0.1:60496)
0928 21:20:43.019022 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.019088 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 21:20:43.020290 22228 catalog_manager.cc:4584] TS 2b0c8d45cb984af5a09ded8e0e834463 (127.5.18.193:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa8d4ed0ee31401db569d910b5168efd: Network error: Client connection negotiation failed: client connection to 127.5.18.193:39299: connect: Connection refused (error 111)
I20260928 21:20:43.024114 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22573
I20260928 21:20:43.029181 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22211
2026-09-28T21:20:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2297 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T21:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:43Z Disabled control of system clock
I20260928 21:20:43.057307 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:33231
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:33231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:43.164844 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:43.165202 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:43.165263 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:43.169690 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:43.169776 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:43.169797 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:43.169817 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:43.169837 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:43.174968 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:33231
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:33231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:43.176221 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:43.177733 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:43.186218 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.186211 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.186273 22748 server_base.cc:1034] running on GCE node
W20260928 21:20:43.186208 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.187031 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:43.188159 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:43.189316 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790630443189300 us; error 33 us; skew 500 ppm
I20260928 21:20:43.191246 22748 webserver.cc:460] Webserver started at http://127.5.18.254:38821/ using document root <none> and password file <none>
I20260928 21:20:43.191825 22748 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:43.191895 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:43.192118 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:43.194253 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "f8cc1fd5e6d242519ad78ee8e35452c7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "72068bdd26dcf7c39a1448cf82d7c50c"
server_key_iv: "eed9f78bb8bea9fc1e46d5b049b14f2e"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.194813 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "f8cc1fd5e6d242519ad78ee8e35452c7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "72068bdd26dcf7c39a1448cf82d7c50c"
server_key_iv: "eed9f78bb8bea9fc1e46d5b049b14f2e"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.198530 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:20:43.201100 22762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.202457 22748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:43.202584 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "f8cc1fd5e6d242519ad78ee8e35452c7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "72068bdd26dcf7c39a1448cf82d7c50c"
server_key_iv: "eed9f78bb8bea9fc1e46d5b049b14f2e"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.202690 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:43.227227 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:43.228022 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:43.228219 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:43.234975 22748 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:33231
I20260928 21:20:43.234998 22814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:33231 every 8 connection(s)
I20260928 21:20:43.235908 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:43.238967 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:43.242995 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22748
I20260928 21:20:43.243100 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:43.243340 5195 external_mini_cluster.cc:1428] Setting key 582ca1f70cf6dde9b03e62e5a8fdef26
I20260928 21:20:43.244776 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: Bootstrap starting.
I20260928 21:20:43.247545 22815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:43.248446 22815 log.cc:824] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:43.250554 22815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: No bootstrap required, opened a new log
I20260928 21:20:43.254142 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.244807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.5.18.254:33231)
0928 21:20:43.245505 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.245525 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.245598 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:20:43.246465 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.246489 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.247002 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.247376 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:43.247885 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.249176 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.250159 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.250876 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.251033 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.251916 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:43.251922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:43.251940 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:43.252457 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:43.252528 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:43.252556 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:43.252653 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.252815 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.253098 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260928 21:20:43.254976 22815 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } }
I20260928 21:20:43.255206 22815 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:43.255261 22815 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cc1fd5e6d242519ad78ee8e35452c7, State: Initialized, Role: FOLLOWER
I20260928 21:20:43.255672 22815 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } }
I20260928 21:20:43.255800 22815 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:43.255861 22815 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:43.255957 22815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:43.256959 22815 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } }
I20260928 21:20:43.257284 22815 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8cc1fd5e6d242519ad78ee8e35452c7; no voters:
I20260928 21:20:43.257596 22815 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:43.257715 22820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:43.257932 22820 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [term 1 LEADER]: Becoming Leader. State: Replica: f8cc1fd5e6d242519ad78ee8e35452c7, State: Running, Role: LEADER
I20260928 21:20:43.258262 22820 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } }
I20260928 21:20:43.258812 22815 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:43.259770 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8cc1fd5e6d242519ad78ee8e35452c7. Latest consensus state: current_term: 1 leader_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } } }
I20260928 21:20:43.259893 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:43.260118 22821 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cc1fd5e6d242519ad78ee8e35452c7" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 33231 } } }
I20260928 21:20:43.260200 22821 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:43.260341 22829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:43.263535 22829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:43.269112 22829 catalog_manager.cc:1347] Generated new cluster ID: 726b0c3358174ae3a8fe501325744a80
I20260928 21:20:43.269204 22829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:43.296973 22829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:43.298518 22829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:43.306102 22829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: Generated new TSK 0
I20260928 21:20:43.307041 22829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:43.317550 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--builtin_ntp_servers=127.5.18.212:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:43.431593 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:43.431910 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:43.431948 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:43.436172 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:43.436249 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:43.436306 22839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:43.441263 22839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:43.442473 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:43.443980 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:43.453992 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.453949 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.453956 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.454115 22839 server_base.cc:1034] running on GCE node
I20260928 21:20:43.454768 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:43.455462 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:43.456624 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790630443456608 us; error 33 us; skew 500 ppm
I20260928 21:20:43.462355 22839 webserver.cc:460] Webserver started at http://127.5.18.193:43563/ using document root <none> and password file <none>
I20260928 21:20:43.463037 22839 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:43.463110 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:43.463320 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:43.465510 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "5f503cbffabf4338b1d21c8fca3bca7c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "16181ab1976c56a90635e3eff00849cf"
server_key_iv: "1f8b7f5b5a97c306675810ee797db8cb"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.466122 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "5f503cbffabf4338b1d21c8fca3bca7c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "16181ab1976c56a90635e3eff00849cf"
server_key_iv: "1f8b7f5b5a97c306675810ee797db8cb"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.469986 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:43.472571 22853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.473985 22839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:43.474112 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "5f503cbffabf4338b1d21c8fca3bca7c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "16181ab1976c56a90635e3eff00849cf"
server_key_iv: "1f8b7f5b5a97c306675810ee797db8cb"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.474221 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:43.495067 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:43.495795 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:43.495968 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:43.496564 22839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:43.497526 22839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:43.497602 22839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.497658 22839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:43.497682 22839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.506059 22839 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:33259
I20260928 21:20:43.506106 22966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:33259 every 8 connection(s)
I20260928 21:20:43.507104 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:43.513918 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22839
I20260928 21:20:43.514022 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:43.514281 5195 external_mini_cluster.cc:1428] Setting key 3c32309bbd467c832c1fc9c5da2263e5
I20260928 21:20:43.516394 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.508944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:51683 (local address 127.5.18.254:33231)
0928 21:20:43.509090 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.509094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.510273 (+ 1179us) server_negotiation.cc:407] Connection header received
0928 21:20:43.511572 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.511575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.511622 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.511694 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:43.512597 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.513133 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.514340 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.514631 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.514728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.515945 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:43.515949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:43.515952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:43.515965 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:43.515989 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:43.515999 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:43.516071 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.516185 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.516247 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:20:43.516379 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--builtin_ntp_servers=127.5.18.212:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:43.517328 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.509407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33231 (local address 127.5.18.193:51683)
0928 21:20:43.510123 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.510165 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.511389 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.511873 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.511880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.511969 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:43.512474 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.512487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.513235 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.513237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.514211 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.514221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.514816 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.514820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.514991 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.515648 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:43.515650 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:43.515705 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:43.515795 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:43.516070 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:43.516075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:43.516079 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.516168 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.516419 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
I20260928 21:20:43.518569 22967 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33231
I20260928 21:20:43.518821 22967 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:43.519450 22967 heartbeater.cc:511] Master 127.5.18.254:33231 requested a full tablet report, sending...
I20260928 21:20:43.521044 22779 ts_manager.cc:194] Registered new tserver with Master: 5f503cbffabf4338b1d21c8fca3bca7c (127.5.18.193:33259)
I20260928 21:20:43.522213 22779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.193:51683
W20260928 21:20:43.626308 22970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:43.626621 22970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:43.626677 22970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:43.630854 22970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:43.630929 22970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:43.631009 22970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:43.635986 22970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:43.637120 22970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:43.638594 22970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:43.648494 22976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.648470 22975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.648505 22978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.648679 22970 server_base.cc:1034] running on GCE node
I20260928 21:20:43.649184 22970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:43.649960 22970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:43.651163 22970 hybrid_clock.cc:648] HybridClock initialized: now 1790630443651123 us; error 53 us; skew 500 ppm
I20260928 21:20:43.653043 22970 webserver.cc:460] Webserver started at http://127.5.18.194:45139/ using document root <none> and password file <none>
I20260928 21:20:43.653648 22970 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:43.653720 22970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:43.653910 22970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:43.656021 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "9f26953ed98b45c2bcf22ec9af1edcd7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "757dcf3d30ab37bb023253e1de1f86c5"
server_key_iv: "9febc83c65d67e0b648eb455006f0339"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.656549 22970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "9f26953ed98b45c2bcf22ec9af1edcd7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "757dcf3d30ab37bb023253e1de1f86c5"
server_key_iv: "9febc83c65d67e0b648eb455006f0339"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.660492 22970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:20:43.663266 22984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.664772 22970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:20:43.664903 22970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "9f26953ed98b45c2bcf22ec9af1edcd7"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "757dcf3d30ab37bb023253e1de1f86c5"
server_key_iv: "9febc83c65d67e0b648eb455006f0339"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.665019 22970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:43.684314 22970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:43.685027 22970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:43.685196 22970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:43.685814 22970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:43.686755 22970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:43.686826 22970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.686869 22970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:43.686889 22970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.695853 22970 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:43159
I20260928 21:20:43.695876 23097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:43159 every 8 connection(s)
I20260928 21:20:43.696847 22970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:43.703037 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 22970
I20260928 21:20:43.703162 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:43.703444 5195 external_mini_cluster.cc:1428] Setting key 5f57e5171a811d91281879cbf435acef
I20260928 21:20:43.705458 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--builtin_ntp_servers=127.5.18.212:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:43.707006 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.698817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:55911 (local address 127.5.18.254:33231)
0928 21:20:43.698960 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.698964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.700195 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 21:20:43.701627 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.701631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.701693 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.701772 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:43.702691 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.703438 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.704716 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.704971 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.705050 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.706474 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:43.706477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:43.706479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:43.706490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:43.706510 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:43.706517 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:43.706575 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.706763 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.706851 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:20:43.707871 23100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.699291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33231 (local address 127.5.18.194:55911)
0928 21:20:43.700025 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.700068 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.701386 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.701957 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.701965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.702056 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:43.702554 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.702566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.703566 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.703569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.704576 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.704588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.705119 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.705123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.705336 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.706125 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:43.706127 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:43.706184 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:43.706281 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:43.706625 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:43.706634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:43.706640 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.706769 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.707106 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260928 21:20:43.708968 23098 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33231
I20260928 21:20:43.709185 23098 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:43.709730 23098 heartbeater.cc:511] Master 127.5.18.254:33231 requested a full tablet report, sending...
I20260928 21:20:43.710795 22779 ts_manager.cc:194] Registered new tserver with Master: 9f26953ed98b45c2bcf22ec9af1edcd7 (127.5.18.194:43159)
I20260928 21:20:43.711542 22779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.194:55911
W20260928 21:20:43.817040 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:43.817329 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:43.817368 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:43.821485 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:43.821557 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:43.821657 23101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:43.826553 23101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:33231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:43.827623 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:43.828941 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:43.838651 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.838703 23101 server_base.cc:1034] running on GCE node
W20260928 21:20:43.838903 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:43.838770 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:43.839416 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:43.840111 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:43.841284 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790630443841262 us; error 34 us; skew 500 ppm
I20260928 21:20:43.843192 23101 webserver.cc:460] Webserver started at http://127.5.18.195:37311/ using document root <none> and password file <none>
I20260928 21:20:43.843745 23101 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:43.843808 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:43.843997 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:43.846158 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "9745e0487c504b8dad738ad42f07115c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "94320be20f7947c59332b0fb4e594a9e"
server_key_iv: "d240f94cd0c4d0262140dc15e04870ca"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.846719 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "9745e0487c504b8dad738ad42f07115c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "94320be20f7947c59332b0fb4e594a9e"
server_key_iv: "d240f94cd0c4d0262140dc15e04870ca"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.850613 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:20:43.853106 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.854372 23101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:43.854501 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "9745e0487c504b8dad738ad42f07115c"
format_stamp: "Formatted at 2026-09-28 21:20:43 on dist-test-slave-rc9f"
server_key: "94320be20f7947c59332b0fb4e594a9e"
server_key_iv: "d240f94cd0c4d0262140dc15e04870ca"
server_key_version: "encryptionkey@0"
I20260928 21:20:43.854625 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:43.876875 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:43.877660 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:43.877835 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:43.878439 23101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:43.879419 23101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:43.879465 23101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.879508 23101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:43.879535 23101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:43.887866 23101 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:37041
I20260928 21:20:43.887890 23228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:37041 every 8 connection(s)
I20260928 21:20:43.888780 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:43.892071 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23101
I20260928 21:20:43.892158 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:43.892385 5195 external_mini_cluster.cc:1428] Setting key be1821c825536defb9189ad1647360b4
I20260928 21:20:43.898511 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.890604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:54717 (local address 127.5.18.254:33231)
0928 21:20:43.890755 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.890759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.891868 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 21:20:43.893411 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.893414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.893478 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.893547 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:43.894717 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.895297 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.896571 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.896836 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.896926 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.898050 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:43.898054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:43.898057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:43.898071 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:43.898092 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:43.898100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:43.898161 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.898297 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.898362 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:20:43.899411 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.890986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:33231 (local address 127.5.18.195:54717)
0928 21:20:43.891691 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.891735 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.893189 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.893762 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.893772 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.893889 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:43.894542 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.894557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.895475 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.895478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.896401 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.896411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.896978 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.896982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.897153 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.897790 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:43.897792 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:43.897834 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:43.897902 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:43.898170 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:43.898178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:43.898184 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.898293 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.898610 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260928 21:20:43.900483 23229 heartbeater.cc:347] Connected to a master server at 127.5.18.254:33231
I20260928 21:20:43.900717 23229 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:43.901249 23229 heartbeater.cc:511] Master 127.5.18.254:33231 requested a full tablet report, sending...
I20260928 21:20:43.902251 22779 ts_manager.cc:194] Registered new tserver with Master: 9745e0487c504b8dad738ad42f07115c (127.5.18.195:37041)
I20260928 21:20:43.902835 22779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.195:54717
I20260928 21:20:43.906100 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:43.911550 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.907968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.5.18.254:33231)
0928 21:20:43.908088 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.908092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.908247 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:20:43.908351 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.908354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.908399 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.908461 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:43.908728 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.909243 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.910078 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.910303 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.910386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.911137 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:43.911142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:43.911144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:43.911157 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:43.911175 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:43.911183 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:43.911240 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.911340 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.911399 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:20:43.914628 22779 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:43.917552 22779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:43.932178 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.926994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.5.18.195:37041)
0928 21:20:43.927410 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.927422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.927448 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:20:43.927519 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.927523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.927746 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.927887 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:43.928180 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929387 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.930934 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931441 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931536 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.931737 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.931897 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.931998 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":88,"threads_started":1}
I20260928 21:20:43.932181 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.926608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:33259 (local address 127.0.0.1:40216)
0928 21:20:43.927637 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.927650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.927721 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.928110 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.928113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.928138 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:43.928226 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.928233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929164 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.929167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.930493 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.930503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931794 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.931882 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.931901 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.931955 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.932019 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":65,"thread_start_us":105,"threads_started":1}
I20260928 21:20:43.932192 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.926187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43159 (local address 127.0.0.1:36376)
0928 21:20:43.926873 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.926928 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.927055 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.928034 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.928037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.928051 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:43.928118 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.928123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929524 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.929527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.930517 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.930526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931579 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.931709 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.931765 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.931836 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.932087 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":149,"thread_start_us":68,"threads_started":1}
I20260928 21:20:43.932425 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.926351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.5.18.194:43159)
0928 21:20:43.927415 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.927427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.927450 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:20:43.927525 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.927530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.927743 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.927890 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:43.928230 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929378 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.930642 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931448 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931970 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.932138 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.932183 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.932273 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":134,"threads_started":1}
I20260928 21:20:43.932549 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.926346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37041 (local address 127.0.0.1:42826)
0928 21:20:43.927054 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:43.927067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:43.927162 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:43.927875 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:43.927881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:43.927901 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:43.928061 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.928068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929490 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.929494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.930826 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:43.930834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931548 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:43.931713 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.931752 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:20:43.931852 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:20:43.931989 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":125,"thread_start_us":117,"threads_started":1}
I20260928 21:20:43.932760 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:43.927307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.5.18.193:33259)
0928 21:20:43.927688 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:43.927698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:43.927719 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:20:43.927822 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:43.927827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:43.927983 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:43.928115 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:43.928327 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.929054 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.930619 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:43.931448 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:43.931549 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:43.931735 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:43.932514 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 21:20:43.932610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":85,"threads_started":1}
I20260928 21:20:43.935511 22901 tablet_service.cc:1467] Processing CreateTablet for tablet 69be188b60d44ed691031b4f91734ac5 (DEFAULT_TABLE table=test-table [id=c31b23c573ba45fb9f8616ad11300a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:43.935544 23032 tablet_service.cc:1467] Processing CreateTablet for tablet 69be188b60d44ed691031b4f91734ac5 (DEFAULT_TABLE table=test-table [id=c31b23c573ba45fb9f8616ad11300a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:43.935544 23163 tablet_service.cc:1467] Processing CreateTablet for tablet 69be188b60d44ed691031b4f91734ac5 (DEFAULT_TABLE table=test-table [id=c31b23c573ba45fb9f8616ad11300a0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:43.936510 22901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69be188b60d44ed691031b4f91734ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:43.936524 23163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69be188b60d44ed691031b4f91734ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:43.936524 23032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69be188b60d44ed691031b4f91734ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:43.943480 23246 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Bootstrap starting.
I20260928 21:20:43.944689 23248 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Bootstrap starting.
I20260928 21:20:43.945686 23246 tablet_bootstrap.cc:654] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:43.945933 23247 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Bootstrap starting.
I20260928 21:20:43.946441 23246 log.cc:824] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:43.946763 23248 tablet_bootstrap.cc:654] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:43.947623 23248 log.cc:824] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:43.948137 23247 tablet_bootstrap.cc:654] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:43.948895 23246 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: No bootstrap required, opened a new log
I20260928 21:20:43.948976 23247 log.cc:824] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:43.949091 23246 ts_tablet_manager.cc:1397] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 21:20:43.949702 23248 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: No bootstrap required, opened a new log
I20260928 21:20:43.949885 23248 ts_tablet_manager.cc:1397] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:20:43.951428 23247 tablet_bootstrap.cc:492] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: No bootstrap required, opened a new log
I20260928 21:20:43.951632 23247 ts_tablet_manager.cc:1397] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:20:43.953155 23248 raft_consensus.cc:348] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.953393 23248 raft_consensus.cc:374] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:43.953449 23248 raft_consensus.cc:729] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9745e0487c504b8dad738ad42f07115c, State: Initialized, Role: FOLLOWER
I20260928 21:20:43.953451 23246 raft_consensus.cc:348] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.953735 23246 raft_consensus.cc:374] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:43.953790 23246 raft_consensus.cc:729] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f503cbffabf4338b1d21c8fca3bca7c, State: Initialized, Role: FOLLOWER
I20260928 21:20:43.953931 23248 consensus_queue.cc:260] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.954224 23246 consensus_queue.cc:260] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.954725 23229 heartbeater.cc:503] Master 127.5.18.254:33231 was elected leader, sending a full tablet report...
I20260928 21:20:43.955242 23246 ts_tablet_manager.cc:1428] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:20:43.955606 22967 heartbeater.cc:503] Master 127.5.18.254:33231 was elected leader, sending a full tablet report...
I20260928 21:20:43.955964 23247 raft_consensus.cc:348] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.956220 23247 raft_consensus.cc:374] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:43.956269 23247 raft_consensus.cc:729] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26953ed98b45c2bcf22ec9af1edcd7, State: Initialized, Role: FOLLOWER
I20260928 21:20:43.956718 23247 consensus_queue.cc:260] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:43.956941 23248 ts_tablet_manager.cc:1428] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:20:43.957705 23098 heartbeater.cc:503] Master 127.5.18.254:33231 was elected leader, sending a full tablet report...
I20260928 21:20:43.957749 23247 ts_tablet_manager.cc:1428] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 21:20:44.008239 22968 tablet.cc:2369] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:44.122987 23253 raft_consensus.cc:482] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:44.123224 23253 raft_consensus.cc:504] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:44.124464 23253 leader_election.cc:288] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26953ed98b45c2bcf22ec9af1edcd7 (127.5.18.194:43159), 9745e0487c504b8dad738ad42f07115c (127.5.18.195:37041)
I20260928 21:20:44.129684 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.124807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:45525 (local address 127.5.18.194:43159)
0928 21:20:44.124935 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:44.124939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:44.125315 (+ 376us) server_negotiation.cc:407] Connection header received
0928 21:20:44.125382 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:44.125385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:44.125435 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:44.125505 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:44.125803 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.126587 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.128311 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.128921 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.129056 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:44.129106 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:44.129440 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 21:20:44.129521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:20:44.129684 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.124853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:56603 (local address 127.5.18.195:37041)
0928 21:20:44.124962 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:44.124966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:44.125198 (+ 232us) server_negotiation.cc:407] Connection header received
0928 21:20:44.125344 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:44.125347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:44.125401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:44.125485 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:44.125815 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.126587 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.128311 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.128952 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.129056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:44.129106 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:44.129440 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 21:20:44.129521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:44.129755 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.124737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:37041 (local address 127.5.18.193:56603)
0928 21:20:44.125094 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:44.125109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:44.125227 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:44.125517 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:44.125520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:44.125582 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:44.125692 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:44.125699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.126720 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.126724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:44.128155 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:44.128170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.129082 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.129087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:44.129284 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:44.129323 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:20:44.129406 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:20:44.129519 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":129,"spinlock_wait_cycles":41728,"thread_start_us":101,"threads_started":1}
I20260928 21:20:44.129927 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.124752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:43159 (local address 127.5.18.193:45525)
0928 21:20:44.125217 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:44.125228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:44.125305 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:44.125517 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:44.125520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:44.125546 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:44.125692 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:44.125699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.126720 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.126724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:44.128175 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:44.128181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.129082 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.129087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:44.129297 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:44.129334 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:20:44.129406 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:20:44.129519 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":240,"thread_start_us":82,"threads_started":1}
I20260928 21:20:44.130482 23183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69be188b60d44ed691031b4f91734ac5" candidate_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9745e0487c504b8dad738ad42f07115c" is_pre_election: true
I20260928 21:20:44.130693 23052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69be188b60d44ed691031b4f91734ac5" candidate_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" is_pre_election: true
I20260928 21:20:44.130903 23183 raft_consensus.cc:2454] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f503cbffabf4338b1d21c8fca3bca7c in term 0.
I20260928 21:20:44.130988 23052 raft_consensus.cc:2454] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f503cbffabf4338b1d21c8fca3bca7c in term 0.
I20260928 21:20:44.131455 22855 leader_election.cc:302] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f503cbffabf4338b1d21c8fca3bca7c, 9745e0487c504b8dad738ad42f07115c; no voters:
I20260928 21:20:44.131719 23253 raft_consensus.cc:2781] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:44.131805 23253 raft_consensus.cc:482] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:44.131847 23253 raft_consensus.cc:3037] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:44.132859 23253 raft_consensus.cc:504] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:44.133184 23253 leader_election.cc:288] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [CANDIDATE]: Term 1 election: Requested vote from peers 9f26953ed98b45c2bcf22ec9af1edcd7 (127.5.18.194:43159), 9745e0487c504b8dad738ad42f07115c (127.5.18.195:37041)
I20260928 21:20:44.133553 23052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69be188b60d44ed691031b4f91734ac5" candidate_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7"
I20260928 21:20:44.133553 23183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69be188b60d44ed691031b4f91734ac5" candidate_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9745e0487c504b8dad738ad42f07115c"
I20260928 21:20:44.133708 23052 raft_consensus.cc:3037] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:44.133749 23183 raft_consensus.cc:3037] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:44.134730 23052 raft_consensus.cc:2454] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f503cbffabf4338b1d21c8fca3bca7c in term 1.
I20260928 21:20:44.134749 23183 raft_consensus.cc:2454] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f503cbffabf4338b1d21c8fca3bca7c in term 1.
I20260928 21:20:44.135035 22855 leader_election.cc:302] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5f503cbffabf4338b1d21c8fca3bca7c, 9745e0487c504b8dad738ad42f07115c, 9f26953ed98b45c2bcf22ec9af1edcd7; no voters:
I20260928 21:20:44.135239 23253 raft_consensus.cc:2781] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:44.135473 23253 raft_consensus.cc:686] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [term 1 LEADER]: Becoming Leader. State: Replica: 5f503cbffabf4338b1d21c8fca3bca7c, State: Running, Role: LEADER
I20260928 21:20:44.135820 23253 consensus_queue.cc:237] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } }
I20260928 21:20:44.139057 22779 catalog_manager.cc:5495] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f503cbffabf4338b1d21c8fca3bca7c (127.5.18.193). New cstate: current_term: 1 leader_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f503cbffabf4338b1d21c8fca3bca7c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 33259 } health_report { overall_health: HEALTHY } } }
W20260928 21:20:44.139655 23230 tablet.cc:2369] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:44.170817 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.167119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.5.18.193:33259)
0928 21:20:44.167261 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:44.167266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:44.167439 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:20:44.167599 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:44.167601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:44.167666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:44.167759 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:44.167973 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.168471 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.169394 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:44.169655 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:44.169739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:44.170065 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:44.170503 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:44.170591 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:44.170656 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:20:44.170704 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:20:44.176991 23183 raft_consensus.cc:1261] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 1 FOLLOWER]: Refusing update from remote peer 5f503cbffabf4338b1d21c8fca3bca7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:44.177734 23052 raft_consensus.cc:1261] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 1 FOLLOWER]: Refusing update from remote peer 5f503cbffabf4338b1d21c8fca3bca7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:44.177845 23257 consensus_queue.cc:1035] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9745e0487c504b8dad738ad42f07115c" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 37041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:44.178355 23253 consensus_queue.cc:1035] T 69be188b60d44ed691031b4f91734ac5 P 5f503cbffabf4338b1d21c8fca3bca7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26953ed98b45c2bcf22ec9af1edcd7" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 43159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:44.193620 22777 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:44.193799 22777 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:44.196367 22777 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8cc1fd5e6d242519ad78ee8e35452c7: Sending DeleteTablet for 3 replicas of tablet 69be188b60d44ed691031b4f91734ac5
I20260928 21:20:44.197239 23032 tablet_service.cc:1514] Processing DeleteTablet for tablet 69be188b60d44ed691031b4f91734ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:44 UTC) from {username='slave'} at 127.0.0.1:36376
I20260928 21:20:44.197240 23163 tablet_service.cc:1514] Processing DeleteTablet for tablet 69be188b60d44ed691031b4f91734ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:44 UTC) from {username='slave'} at 127.0.0.1:42826
I20260928 21:20:44.197463 22901 tablet_service.cc:1514] Processing DeleteTablet for tablet 69be188b60d44ed691031b4f91734ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:44 UTC) from {username='slave'} at 127.0.0.1:40216
I20260928 21:20:44.197822 23271 tablet_replica.cc:330] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: stopping tablet replica
I20260928 21:20:44.198161 23270 tablet_replica.cc:330] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: stopping tablet replica
I20260928 21:20:44.198212 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22839
I20260928 21:20:44.198809 23270 raft_consensus.cc:2229] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:44.199180 23270 raft_consensus.cc:2258] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:44.200224 23271 raft_consensus.cc:2229] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:44.200500 23271 raft_consensus.cc:2258] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:44.200853 23270 ts_tablet_manager.cc:1905] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:44.201972 23271 ts_tablet_manager.cc:1905] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:44.203269 23270 ts_tablet_manager.cc:1918] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:44.203359 23270 log.cc:1196] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/69be188b60d44ed691031b4f91734ac5
I20260928 21:20:44.203657 23270 ts_tablet_manager.cc:1939] T 69be188b60d44ed691031b4f91734ac5 P 9745e0487c504b8dad738ad42f07115c: Deleting consensus metadata
I20260928 21:20:44.204495 23271 ts_tablet_manager.cc:1918] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:44.204589 23271 log.cc:1196] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/69be188b60d44ed691031b4f91734ac5
I20260928 21:20:44.204823 22764 catalog_manager.cc:4841] TS 9745e0487c504b8dad738ad42f07115c (127.5.18.195:37041): tablet 69be188b60d44ed691031b4f91734ac5 (table test-table [id=c31b23c573ba45fb9f8616ad11300a0c]) successfully deleted
I20260928 21:20:44.204877 23271 ts_tablet_manager.cc:1939] T 69be188b60d44ed691031b4f91734ac5 P 9f26953ed98b45c2bcf22ec9af1edcd7: Deleting consensus metadata
W20260928 21:20:44.205892 22765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:33259 (error 108)
I20260928 21:20:44.206177 22763 catalog_manager.cc:4841] TS 9f26953ed98b45c2bcf22ec9af1edcd7 (127.5.18.194:43159): tablet 69be188b60d44ed691031b4f91734ac5 (table test-table [id=c31b23c573ba45fb9f8616ad11300a0c]) successfully deleted
I20260928 21:20:44.206656 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22970
I20260928 21:20:44.206921 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:44.206660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:33259 (local address 127.0.0.1:40240)
0928 21:20:44.206782 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:44.206843 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:33259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:20:44.207826 22765 catalog_manager.cc:4584] TS 5f503cbffabf4338b1d21c8fca3bca7c (127.5.18.193:33259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69be188b60d44ed691031b4f91734ac5: Network error: Client connection negotiation failed: client connection to 127.5.18.193:33259: connect: Connection refused (error 111)
I20260928 21:20:44.211615 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23101
I20260928 21:20:44.216395 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 22748
2026-09-28T21:20:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1186 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:44 dist-test-slave-rc9f krb5kdc[23276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:44 dist-test-slave-rc9f krb5kdc[23276](info): set up 2 sockets
Sep 28 21:20:44 dist-test-slave-rc9f krb5kdc[23276](info): commencing operation
krb5kdc: starting...
W20260928 21:20:45.173163 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:20:45.292927 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43331
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:45.404218 23289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:45.404496 23289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:45.404536 23289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:45.408651 23289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:45.408720 23289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:45.408737 23289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:45.408753 23289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:45.408767 23289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:45.414120 23289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:45.415253 23289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:45.416287 23289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:45.424676 23297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:45.424770 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:45.424813 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:45.424863 23289 server_base.cc:1034] running on GCE node
I20260928 21:20:45.425518 23289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:45.426649 23289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:45.427817 23289 hybrid_clock.cc:648] HybridClock initialized: now 1790630445427786 us; error 48 us; skew 500 ppm
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:45.430485 23289 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:20:45.431360 23289 webserver.cc:460] Webserver started at http://127.5.18.254:44995/ using document root <none> and password file <none>
I20260928 21:20:45.431896 23289 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:45.431958 23289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:45.432156 23289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:45.434280 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "e1e941d5f84346ad81a8d6a706d548d0"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b596658e392480447e7cc2a2b31d5d1f"
server_key_iv: "509b3969746a043e69aadbe7a25a519d"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.434827 23289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "e1e941d5f84346ad81a8d6a706d548d0"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b596658e392480447e7cc2a2b31d5d1f"
server_key_iv: "509b3969746a043e69aadbe7a25a519d"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.438617 23289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:45.441015 23304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.442274 23289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:45.442390 23289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "e1e941d5f84346ad81a8d6a706d548d0"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b596658e392480447e7cc2a2b31d5d1f"
server_key_iv: "509b3969746a043e69aadbe7a25a519d"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.442494 23289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:45.458703 23289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:45.461763 23289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:45.461944 23289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:45.468578 23289 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:43331
I20260928 21:20:45.468591 23356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:43331 every 8 connection(s)
I20260928 21:20:45.469527 23289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:45.472244 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:45.477655 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: Bootstrap starting.
I20260928 21:20:45.479241 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23289
I20260928 21:20:45.479368 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:45.479677 5195 external_mini_cluster.cc:1428] Setting key 9fbc4fa4130eaa6e5456e88899377735
I20260928 21:20:45.480363 23357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:45.481360 23357 log.cc:824] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:45.483487 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: No bootstrap required, opened a new log
I20260928 21:20:45.487053 23357 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } }
I20260928 21:20:45.487320 23357 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:45.487378 23357 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e941d5f84346ad81a8d6a706d548d0, State: Initialized, Role: FOLLOWER
I20260928 21:20:45.487803 23357 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } }
I20260928 21:20:45.487926 23357 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:45.487988 23357 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:45.488088 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:45.489295 23357 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } }
I20260928 21:20:45.489660 23357 leader_election.cc:302] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e941d5f84346ad81a8d6a706d548d0; no voters:
I20260928 21:20:45.490021 23357 leader_election.cc:288] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:45.490130 23362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:45.490341 23362 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [term 1 LEADER]: Becoming Leader. State: Replica: e1e941d5f84346ad81a8d6a706d548d0, State: Running, Role: LEADER
I20260928 21:20:45.490670 23362 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } }
I20260928 21:20:45.491374 23357 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:45.491911 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e941d5f84346ad81a8d6a706d548d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } } }
I20260928 21:20:45.492033 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:45.492089 23363 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e941d5f84346ad81a8d6a706d548d0. Latest consensus state: current_term: 1 leader_uuid: "e1e941d5f84346ad81a8d6a706d548d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e941d5f84346ad81a8d6a706d548d0" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 43331 } } }
I20260928 21:20:45.492182 23363 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:45.492419 23367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:45.494724 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:45.480842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.5.18.254:43331)
0928 21:20:45.481290 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:45.481305 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:45.481341 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:20:45.482328 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:45.482355 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:45.482788 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:45.483119 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:45.483936 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.484917 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.485828 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.486333 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.486481 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:45.489267 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:45.489273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:45.489288 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:45.489316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:45.491390 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.491888 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.491894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.491900 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.492060 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.492378 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.492381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.492383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.492874 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:45.493063 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:45.493489 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 21:20:45.493816 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260928 21:20:45.495354 23367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:45.500998 23367 catalog_manager.cc:1347] Generated new cluster ID: d392215235b2420eb3d76f0ae0d9baf8
I20260928 21:20:45.501070 23367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:45.524966 23367 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:45.526046 23367 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:45.530977 23367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: Generated new TSK 0
I20260928 21:20:45.531688 23367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:20:45.565006 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:45.674666 23383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:45.674969 23383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:45.675014 23383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:45.679106 23383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:45.679172 23383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:45.679239 23383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:45.684508 23383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:45.685667 23383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:45.686687 23383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:45.696352 23388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:45.696445 23383 server_base.cc:1034] running on GCE node
W20260928 21:20:45.696382 23391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:45.696352 23389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:45.697024 23383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:45.697712 23383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:45.698889 23383 hybrid_clock.cc:648] HybridClock initialized: now 1790630445698865 us; error 33 us; skew 500 ppm
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:45.701687 23383 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:20:45.702584 23383 webserver.cc:460] Webserver started at http://127.5.18.193:35341/ using document root <none> and password file <none>
I20260928 21:20:45.703145 23383 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:45.703210 23383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:45.703424 23383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:45.705449 23383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "e32139a30fea42f28aa90442fe4849a7"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "4cf263cc332ec57b3d96c330d9a56edd"
server_key_iv: "63e762b9013a3804b6172960d8e50474"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.706018 23383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "e32139a30fea42f28aa90442fe4849a7"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "4cf263cc332ec57b3d96c330d9a56edd"
server_key_iv: "63e762b9013a3804b6172960d8e50474"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.709666 23383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:45.712014 23398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.713238 23383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:45.713346 23383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "e32139a30fea42f28aa90442fe4849a7"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "4cf263cc332ec57b3d96c330d9a56edd"
server_key_iv: "63e762b9013a3804b6172960d8e50474"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.713442 23383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:45.744470 23383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:45.747622 23383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:45.747805 23383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:45.748377 23383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:45.749168 23383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:45.749213 23383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.749264 23383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:45.749293 23383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.757725 23383 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:39105
I20260928 21:20:45.757786 23511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:39105 every 8 connection(s)
I20260928 21:20:45.758644 23383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:45.761777 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23383
I20260928 21:20:45.761886 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:45.762168 5195 external_mini_cluster.cc:1428] Setting key 66d849e61904ef5117bce91af38f44f7
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:20:45.772104 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:45.760425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:54289 (local address 127.5.18.254:43331)
0928 21:20:45.760545 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:45.760548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:45.761424 (+ 876us) server_negotiation.cc:407] Connection header received
0928 21:20:45.762714 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:45.762717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:45.762765 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:45.762830 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:45.764029 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.764565 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.765697 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.765930 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.766013 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:45.768751 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:45.768755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:45.768757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:45.768782 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:45.770106 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.770672 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.770674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.770675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.770739 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.771087 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.771090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.771091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.771265 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:45.771339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:45.771850 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 21:20:45.771961 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:20:45.773001 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:45.760814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43331 (local address 127.5.18.193:54289)
0928 21:20:45.761255 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:45.761300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:45.762508 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:45.762984 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:45.762991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:45.763379 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:45.763888 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:45.763902 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.764682 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.764685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:45.765558 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:45.765592 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.766062 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.766066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:45.766230 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:45.766845 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:45.766862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:45.768579 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:45.770218 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:45.770225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:45.770241 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:45.770577 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:45.770814 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:45.770817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:45.770819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:45.771004 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:45.771376 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:45.771381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:45.771636 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 21:20:45.771823 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:20:45.772133 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:20:45.774184 23512 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43331
I20260928 21:20:45.774437 23512 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:45.774907 23512 heartbeater.cc:511] Master 127.5.18.254:43331 requested a full tablet report, sending...
I20260928 21:20:45.776345 23321 ts_manager.cc:194] Registered new tserver with Master: e32139a30fea42f28aa90442fe4849a7 (127.5.18.193:39105)
I20260928 21:20:45.777478 23321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:54289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:20:45.788654 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:45.898662 23517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:45.898955 23517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:45.899017 23517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:45.903153 23517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:45.903226 23517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:45.903308 23517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:45.908752 23517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:45.909912 23517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:45.911000 23517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:45.920670 23525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:45.920740 23517 server_base.cc:1034] running on GCE node
W20260928 21:20:45.920697 23522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:45.920693 23523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:45.921239 23517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:45.921873 23517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:45.923023 23517 hybrid_clock.cc:648] HybridClock initialized: now 1790630445922997 us; error 38 us; skew 500 ppm
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:45.925726 23517 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:20:45.926573 23517 webserver.cc:460] Webserver started at http://127.5.18.194:38435/ using document root <none> and password file <none>
I20260928 21:20:45.927117 23517 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:45.927184 23517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:45.927376 23517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:45.929396 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "c5bcf36a7df84c1a8d6079e952309363"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b598832fa916ed7e0f116a8410c680ea"
server_key_iv: "5ddacc97a6225c01ab34d811b0aa3a05"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.929991 23517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "c5bcf36a7df84c1a8d6079e952309363"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b598832fa916ed7e0f116a8410c680ea"
server_key_iv: "5ddacc97a6225c01ab34d811b0aa3a05"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.933825 23517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:45.936368 23532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.937701 23517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:45.937841 23517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "c5bcf36a7df84c1a8d6079e952309363"
format_stamp: "Formatted at 2026-09-28 21:20:45 on dist-test-slave-rc9f"
server_key: "b598832fa916ed7e0f116a8410c680ea"
server_key_iv: "5ddacc97a6225c01ab34d811b0aa3a05"
server_key_version: "encryptionkey@0"
I20260928 21:20:45.937952 23517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:45.950665 23517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:45.953470 23517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:45.953661 23517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:45.954231 23517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:45.955031 23517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:45.955075 23517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.955127 23517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:45.955153 23517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:45.963258 23517 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:34245
I20260928 21:20:45.963284 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:34245 every 8 connection(s)
I20260928 21:20:45.964148 23517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:45.964674 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23517
I20260928 21:20:45.964813 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:45.965123 5195 external_mini_cluster.cc:1428] Setting key 9fb2a905833cc754253b40ae3aecaac0
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
Sep 28 21:20:45 dist-test-slave-rc9f krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:45.981350 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:45.966069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:36913 (local address 127.5.18.254:43331)
0928 21:20:45.966201 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:45.966205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:45.967204 (+ 999us) server_negotiation.cc:407] Connection header received
0928 21:20:45.968777 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:45.968780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:45.968837 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:45.968922 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:45.970135 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.970948 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.972050 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.972382 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.972491 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:45.977481 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:45.977485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:45.977487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:45.977516 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:45.979061 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.979705 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.979708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.979710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.979817 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:45.980185 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:45.980187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:45.980189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:45.980373 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:45.980460 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:45.981047 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 21:20:45.981182 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:45.982273 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:45.966532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43331 (local address 127.5.18.194:36913)
0928 21:20:45.967033 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:45.967094 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:45.968565 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:45.969087 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:45.969094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:45.969457 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:45.969996 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:45.970010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.971063 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.971066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:45.971936 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:45.971945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:45.972548 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:45.972551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:45.972709 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:45.973334 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:45.973350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:45.977286 (+ 3936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:45.979192 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:45.979199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:45.979219 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:45.979580 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:45.979909 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:45.979911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:45.979913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:45.980104 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:45.980524 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:45.980530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:45.980802 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 21:20:45.980987 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 21:20:45.981355 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260928 21:20:45.983665 23646 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43331
I20260928 21:20:45.983917 23646 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:45.984432 23646 heartbeater.cc:511] Master 127.5.18.254:43331 requested a full tablet report, sending...
I20260928 21:20:45.985513 23321 ts_manager.cc:194] Registered new tserver with Master: c5bcf36a7df84c1a8d6079e952309363 (127.5.18.194:34245)
I20260928 21:20:45.986109 23321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:36913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:20:45.992973 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:46.101472 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:46.101780 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:46.101822 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:46.105922 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:46.105983 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:46.106051 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:46.111428 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:46.112555 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:46.113663 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:46.123873 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:46.123888 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:46.123888 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:46.124119 23651 server_base.cc:1034] running on GCE node
I20260928 21:20:46.124795 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:46.125447 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:46.126626 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790630446126598 us; error 41 us; skew 500 ppm
Sep 28 21:20:46 dist-test-slave-rc9f krb5kdc[23276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630446, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:46.129503 23651 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:20:46.130430 23651 webserver.cc:460] Webserver started at http://127.5.18.195:36361/ using document root <none> and password file <none>
I20260928 21:20:46.130976 23651 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:46.131035 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:46.131201 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:46.133260 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "909aed05af2441b7b868ee39a487fa1e"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "a0d7b030f300b35286e6341e3a18fb95"
server_key_iv: "da23ad80482cd868ebbc9eaca4eda7c5"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.133841 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "909aed05af2441b7b868ee39a487fa1e"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "a0d7b030f300b35286e6341e3a18fb95"
server_key_iv: "da23ad80482cd868ebbc9eaca4eda7c5"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.137671 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 21:20:46.140090 23666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.141390 23651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:46.141492 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "909aed05af2441b7b868ee39a487fa1e"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "a0d7b030f300b35286e6341e3a18fb95"
server_key_iv: "da23ad80482cd868ebbc9eaca4eda7c5"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.141645 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:46.161293 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:46.164245 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:46.164402 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:46.164945 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:46.165781 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:46.165838 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.165877 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:46.165897 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.174226 23651 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:41537
I20260928 21:20:46.174266 23779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:41537 every 8 connection(s)
I20260928 21:20:46.175122 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:46.179095 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23651
I20260928 21:20:46.179214 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:46.179514 5195 external_mini_cluster.cc:1428] Setting key 8afd9a1ad92a9978accc1e341032d1bf
Sep 28 21:20:46 dist-test-slave-rc9f krb5kdc[23276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630446, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:46.188645 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.176855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:37305 (local address 127.5.18.254:43331)
0928 21:20:46.177014 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.177018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.177879 (+ 861us) server_negotiation.cc:407] Connection header received
0928 21:20:46.179119 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.179123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.179186 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.179283 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:46.180639 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.181399 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.182612 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.182832 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.182919 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.185349 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:46.185352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:46.185355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:46.185380 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:46.186597 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:46.187169 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:46.187183 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:46.187184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:46.187264 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:46.187576 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:46.187580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:46.187581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:46.187758 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:46.187843 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.188353 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.188478 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 21:20:46.189462 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.177250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:43331 (local address 127.5.18.195:37305)
0928 21:20:46.177720 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.177779 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.178908 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.179474 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.179481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.179935 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:46.180501 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.180518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.181512 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.181516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.182486 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.182496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.182970 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.182974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.183139 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.183720 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:46.183736 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:46.185177 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:46.186704 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:46.186710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:46.186727 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:46.187070 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:46.187345 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:46.187348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:46.187350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:46.187491 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:46.187862 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:46.187868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:46.188119 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.188286 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.188616 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260928 21:20:46.190575 23780 heartbeater.cc:347] Connected to a master server at 127.5.18.254:43331
I20260928 21:20:46.190814 23780 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:46.191299 23780 heartbeater.cc:511] Master 127.5.18.254:43331 requested a full tablet report, sending...
I20260928 21:20:46.192281 23321 ts_manager.cc:194] Registered new tserver with Master: 909aed05af2441b7b868ee39a487fa1e (127.5.18.195:41537)
I20260928 21:20:46.192951 23321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:37305
I20260928 21:20:46.193255 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:46.201660 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.195138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.5.18.254:43331)
0928 21:20:46.195268 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.195272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.195409 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:20:46.195496 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.195498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.195545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.195641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:46.196250 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.196745 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.197455 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.197698 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.197768 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.199132 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:46.199135 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:46.199137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:46.199157 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:46.200256 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:46.200677 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:46.200680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:46.200681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:46.200748 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:46.200965 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:46.200967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:46.200968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:46.201143 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:46.201252 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.201407 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.201481 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:20:46.204366 23321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:46.207365 23321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:46.221799 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.216489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.5.18.194:34245)
0928 21:20:46.217188 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.217201 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.217224 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:20:46.217294 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.217299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.217449 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.217637 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:46.217802 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.218652 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.220441 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221029 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.221151 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.221319 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.221492 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.221624 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":105,"threads_started":1}
I20260928 21:20:46.221830 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.215520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39105 (local address 127.0.0.1:56790)
0928 21:20:46.216035 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.216084 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.216217 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.217107 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.217113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.217135 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:46.217330 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.217340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.218992 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.218996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.220312 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.220319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221349 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.221352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.221439 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.221461 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.221512 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.221654 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":140,"thread_start_us":93,"threads_started":1}
I20260928 21:20:46.221809 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.216256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:34245 (local address 127.0.0.1:48228)
0928 21:20:46.216551 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.216565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.216655 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.217603 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.217606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.217624 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:46.217708 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.217713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.218784 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.218788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.220282 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.220294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221161 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.221165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.221288 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.221328 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.221440 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.221641 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":40,"thread_start_us":94,"threads_started":1}
I20260928 21:20:46.222157 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.216598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41546 (local address 127.5.18.195:41537)
0928 21:20:46.216965 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.216979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.217004 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:20:46.217097 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.217103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.217305 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.217462 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:46.217890 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.219025 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.220371 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221212 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.221317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.221472 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.221923 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.222021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":173,"threads_started":1}
I20260928 21:20:46.222369 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.215662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:41537 (local address 127.0.0.1:41546)
0928 21:20:46.216171 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.216760 (+ 589us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.216869 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.217661 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.217664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.217683 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:46.217784 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.217791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.219144 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.219147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.220251 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.220261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221526 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.221530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.221689 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.221711 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.221774 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.221832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":79,"thread_start_us":107,"threads_started":1}
I20260928 21:20:46.222635 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.215650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.5.18.193:39105)
0928 21:20:46.216369 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.216386 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.216410 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:20:46.216485 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.216491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.216702 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.217477 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:46.217653 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.218858 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.220405 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.221212 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.222216 (+ 1004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.222361 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.222412 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.222505 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":72,"threads_started":1}
I20260928 21:20:46.225519 23714 tablet_service.cc:1467] Processing CreateTablet for tablet 83a085c6342842e9a219a9186aa6367b (DEFAULT_TABLE table=test-table [id=ddd3b53fa9704edb91a4c6afae2f816b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:46.225485 23580 tablet_service.cc:1467] Processing CreateTablet for tablet 83a085c6342842e9a219a9186aa6367b (DEFAULT_TABLE table=test-table [id=ddd3b53fa9704edb91a4c6afae2f816b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:46.226699 23714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a085c6342842e9a219a9186aa6367b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:46.226696 23580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a085c6342842e9a219a9186aa6367b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:46.227257 23446 tablet_service.cc:1467] Processing CreateTablet for tablet 83a085c6342842e9a219a9186aa6367b (DEFAULT_TABLE table=test-table [id=ddd3b53fa9704edb91a4c6afae2f816b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:46.228096 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83a085c6342842e9a219a9186aa6367b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:46.234634 23798 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Bootstrap starting.
I20260928 21:20:46.234946 23797 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Bootstrap starting.
I20260928 21:20:46.236994 23797 tablet_bootstrap.cc:654] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:46.237397 23798 tablet_bootstrap.cc:654] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:46.237862 23799 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Bootstrap starting.
I20260928 21:20:46.238255 23797 log.cc:824] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:46.238247 23798 log.cc:824] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:46.239936 23799 tablet_bootstrap.cc:654] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:46.240162 23797 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: No bootstrap required, opened a new log
I20260928 21:20:46.240341 23797 ts_tablet_manager.cc:1397] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 21:20:46.240574 23798 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: No bootstrap required, opened a new log
I20260928 21:20:46.240763 23798 ts_tablet_manager.cc:1397] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:20:46.241206 23799 log.cc:824] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:46.243341 23799 tablet_bootstrap.cc:492] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: No bootstrap required, opened a new log
I20260928 21:20:46.243541 23799 ts_tablet_manager.cc:1397] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:20:46.243801 23797 raft_consensus.cc:348] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.244035 23797 raft_consensus.cc:374] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:46.244112 23797 raft_consensus.cc:729] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32139a30fea42f28aa90442fe4849a7, State: Initialized, Role: FOLLOWER
I20260928 21:20:46.244587 23797 consensus_queue.cc:260] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.245052 23798 raft_consensus.cc:348] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.245393 23798 raft_consensus.cc:374] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:46.245455 23798 raft_consensus.cc:729] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5bcf36a7df84c1a8d6079e952309363, State: Initialized, Role: FOLLOWER
I20260928 21:20:46.245814 23797 ts_tablet_manager.cc:1428] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:20:46.245947 23798 consensus_queue.cc:260] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.246172 23512 heartbeater.cc:503] Master 127.5.18.254:43331 was elected leader, sending a full tablet report...
I20260928 21:20:46.246630 23646 heartbeater.cc:503] Master 127.5.18.254:43331 was elected leader, sending a full tablet report...
I20260928 21:20:46.246912 23798 ts_tablet_manager.cc:1428] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:20:46.247303 23799 raft_consensus.cc:348] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.247556 23799 raft_consensus.cc:374] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:46.247617 23799 raft_consensus.cc:729] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909aed05af2441b7b868ee39a487fa1e, State: Initialized, Role: FOLLOWER
I20260928 21:20:46.248014 23799 consensus_queue.cc:260] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.248610 23780 heartbeater.cc:503] Master 127.5.18.254:43331 was elected leader, sending a full tablet report...
I20260928 21:20:46.248939 23799 ts_tablet_manager.cc:1428] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 21:20:46.259845 23513 tablet.cc:2369] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:46.417537 23805 raft_consensus.cc:482] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:46.417821 23805 raft_consensus.cc:504] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.418941 23805 leader_election.cc:288] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e32139a30fea42f28aa90442fe4849a7 (127.5.18.193:39105), c5bcf36a7df84c1a8d6079e952309363 (127.5.18.194:34245)
I20260928 21:20:46.423843 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.419316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:60639 (local address 127.5.18.194:34245)
0928 21:20:46.419427 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.419431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.419655 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:20:46.419745 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.419747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.419802 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.419868 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:46.420088 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.420824 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.422569 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.423228 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.423355 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.423410 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.423642 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.423713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:20:46.423843 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.419251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:33583 (local address 127.5.18.193:39105)
0928 21:20:46.419381 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.419386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.419474 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:20:46.419621 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.419624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.419679 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.419753 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:46.420014 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.420803 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.422602 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.423261 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.423372 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.423412 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.423648 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.423713 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:20:46.423899 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.419201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39105 (local address 127.5.18.195:33583)
0928 21:20:46.419399 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.419413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.419514 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.419779 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.419781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.419793 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:46.419917 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.419923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.420924 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.420927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.422463 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.422470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.423369 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.423373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.423503 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.423548 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.423620 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.423730 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":162}
I20260928 21:20:46.423907 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.419200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:34245 (local address 127.5.18.195:60639)
0928 21:20:46.419568 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.419579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.419668 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.419880 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.419883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.419896 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:46.419994 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.419999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.420924 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.420927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.422435 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.422447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.423369 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.423373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.423499 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.423545 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.423619 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.423730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":178,"thread_start_us":114,"threads_started":1}
I20260928 21:20:46.424598 23600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a085c6342842e9a219a9186aa6367b" candidate_uuid: "909aed05af2441b7b868ee39a487fa1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bcf36a7df84c1a8d6079e952309363" is_pre_election: true
I20260928 21:20:46.424597 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a085c6342842e9a219a9186aa6367b" candidate_uuid: "909aed05af2441b7b868ee39a487fa1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32139a30fea42f28aa90442fe4849a7" is_pre_election: true
I20260928 21:20:46.424919 23600 raft_consensus.cc:2454] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909aed05af2441b7b868ee39a487fa1e in term 0.
I20260928 21:20:46.424919 23466 raft_consensus.cc:2454] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909aed05af2441b7b868ee39a487fa1e in term 0.
I20260928 21:20:46.425420 23667 leader_election.cc:302] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 909aed05af2441b7b868ee39a487fa1e, c5bcf36a7df84c1a8d6079e952309363, e32139a30fea42f28aa90442fe4849a7; no voters:
I20260928 21:20:46.425694 23805 raft_consensus.cc:2781] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:46.425786 23805 raft_consensus.cc:482] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:46.425817 23805 raft_consensus.cc:3037] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 0 FOLLOWER]: Advancing to term 1
W20260928 21:20:46.425980 23781 tablet.cc:2369] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:46.426976 23805 raft_consensus.cc:504] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.427299 23805 leader_election.cc:288] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [CANDIDATE]: Term 1 election: Requested vote from peers e32139a30fea42f28aa90442fe4849a7 (127.5.18.193:39105), c5bcf36a7df84c1a8d6079e952309363 (127.5.18.194:34245)
I20260928 21:20:46.427666 23600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a085c6342842e9a219a9186aa6367b" candidate_uuid: "909aed05af2441b7b868ee39a487fa1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5bcf36a7df84c1a8d6079e952309363"
I20260928 21:20:46.427666 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83a085c6342842e9a219a9186aa6367b" candidate_uuid: "909aed05af2441b7b868ee39a487fa1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32139a30fea42f28aa90442fe4849a7"
I20260928 21:20:46.427783 23600 raft_consensus.cc:3037] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:46.427778 23466 raft_consensus.cc:3037] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:46.428730 23600 raft_consensus.cc:2454] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909aed05af2441b7b868ee39a487fa1e in term 1.
I20260928 21:20:46.428750 23466 raft_consensus.cc:2454] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909aed05af2441b7b868ee39a487fa1e in term 1.
I20260928 21:20:46.429031 23668 leader_election.cc:302] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909aed05af2441b7b868ee39a487fa1e, e32139a30fea42f28aa90442fe4849a7; no voters:
I20260928 21:20:46.429185 23805 raft_consensus.cc:2781] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:46.429379 23805 raft_consensus.cc:686] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 1 LEADER]: Becoming Leader. State: Replica: 909aed05af2441b7b868ee39a487fa1e, State: Running, Role: LEADER
I20260928 21:20:46.429745 23805 consensus_queue.cc:237] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } }
I20260928 21:20:46.432660 23320 catalog_manager.cc:5495] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 909aed05af2441b7b868ee39a487fa1e (127.5.18.195). New cstate: current_term: 1 leader_uuid: "909aed05af2441b7b868ee39a487fa1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909aed05af2441b7b868ee39a487fa1e" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 } health_report { overall_health: UNKNOWN } } }
I20260928 21:20:46.464356 23796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.460866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.5.18.195:41537)
0928 21:20:46.461000 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.461004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.461018 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:20:46.461096 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.461099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.461154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.461244 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:46.461495 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.462030 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.462878 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.463089 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.463173 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.463501 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:46.463925 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:46.464055 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.464162 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.464202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:20:46.465355 23647 tablet.cc:2369] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:46.470356 23600 raft_consensus.cc:1261] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 1 FOLLOWER]: Refusing update from remote peer 909aed05af2441b7b868ee39a487fa1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:46.470423 23466 raft_consensus.cc:1261] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 1 FOLLOWER]: Refusing update from remote peer 909aed05af2441b7b868ee39a487fa1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:46.471066 23805 consensus_queue.cc:1035] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5bcf36a7df84c1a8d6079e952309363" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:46.471266 23807 consensus_queue.cc:1035] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32139a30fea42f28aa90442fe4849a7" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:46.478752 23810 mvcc.cc:205] Tried to move back new op lower bound from 7334422308735008768 to 7334422308578492416. Current Snapshot: MvccSnapshot[applied={T|T < 7334422308735008768}]
I20260928 21:20:46.488343 23320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:46.488508 23320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:46.490856 23320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e941d5f84346ad81a8d6a706d548d0: Sending DeleteTablet for 3 replicas of tablet 83a085c6342842e9a219a9186aa6367b
I20260928 21:20:46.491708 23446 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a085c6342842e9a219a9186aa6367b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:46 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:56790
I20260928 21:20:46.491842 23714 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a085c6342842e9a219a9186aa6367b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:46 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:41546
I20260928 21:20:46.491839 23580 tablet_service.cc:1514] Processing DeleteTablet for tablet 83a085c6342842e9a219a9186aa6367b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:46 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:48228
I20260928 21:20:46.492317 23821 tablet_replica.cc:330] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7: stopping tablet replica
I20260928 21:20:46.492686 23821 raft_consensus.cc:2229] T 83a085c6342842e9a219a9186aa6367b P e32139a30fea42f28aa90442fe4849a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:46.492956 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23383
I20260928 21:20:46.494415 23820 tablet_replica.cc:330] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: stopping tablet replica
I20260928 21:20:46.494691 23820 raft_consensus.cc:2229] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:46.495091 23820 raft_consensus.cc:2258] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:46.495594 23822 tablet_replica.cc:330] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: stopping tablet replica
I20260928 21:20:46.495880 23822 raft_consensus.cc:2229] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:46.496171 23822 raft_consensus.cc:2258] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:46.497718 23822 ts_tablet_manager.cc:1905] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:46.498528 23820 ts_tablet_manager.cc:1905] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:46.500371 23822 ts_tablet_manager.cc:1918] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:46.500466 23822 log.cc:1196] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/83a085c6342842e9a219a9186aa6367b
I20260928 21:20:46.500751 23822 ts_tablet_manager.cc:1939] T 83a085c6342842e9a219a9186aa6367b P c5bcf36a7df84c1a8d6079e952309363: Deleting consensus metadata
W20260928 21:20:46.501248 23306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:39105 (error 108)
I20260928 21:20:46.501403 23820 ts_tablet_manager.cc:1918] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:46.501545 23820 log.cc:1196] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/83a085c6342842e9a219a9186aa6367b
I20260928 21:20:46.501941 23820 ts_tablet_manager.cc:1939] T 83a085c6342842e9a219a9186aa6367b P 909aed05af2441b7b868ee39a487fa1e: Deleting consensus metadata
I20260928 21:20:46.502075 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23517
I20260928 21:20:46.502076 23305 catalog_manager.cc:4841] TS c5bcf36a7df84c1a8d6079e952309363 (127.5.18.194:34245): tablet 83a085c6342842e9a219a9186aa6367b (table test-table [id=ddd3b53fa9704edb91a4c6afae2f816b]) successfully deleted
I20260928 21:20:46.503211 23305 catalog_manager.cc:4841] TS 909aed05af2441b7b868ee39a487fa1e (127.5.18.195:41537): tablet 83a085c6342842e9a219a9186aa6367b (table test-table [id=ddd3b53fa9704edb91a4c6afae2f816b]) successfully deleted
I20260928 21:20:46.504483 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.504174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:39105 (local address 127.0.0.1:56798)
0928 21:20:46.504317 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.504396 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:39105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 21:20:46.504666 23306 catalog_manager.cc:4584] TS e32139a30fea42f28aa90442fe4849a7 (127.5.18.193:39105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83a085c6342842e9a219a9186aa6367b: Network error: Client connection negotiation failed: client connection to 127.5.18.193:39105: connect: Connection refused (error 111)
I20260928 21:20:46.507597 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23651
I20260928 21:20:46.512382 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23289
2026-09-28T21:20:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2296 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T21:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:46Z Disabled control of system clock
I20260928 21:20:46.539914 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37857
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:46.648108 23827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:46.648389 23827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:46.648424 23827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:46.652673 23827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:46.652750 23827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:46.652765 23827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:46.652781 23827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:46.652796 23827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:46.657786 23827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:37857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:46.658864 23827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:46.660228 23827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:46.668465 23832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:46.668558 23827 server_base.cc:1034] running on GCE node
W20260928 21:20:46.668448 23835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:46.668473 23833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:46.669323 23827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:46.670502 23827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:46.671661 23827 hybrid_clock.cc:648] HybridClock initialized: now 1790630446671642 us; error 35 us; skew 500 ppm
I20260928 21:20:46.673434 23827 webserver.cc:460] Webserver started at http://127.5.18.254:43735/ using document root <none> and password file <none>
I20260928 21:20:46.674015 23827 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:46.674086 23827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:46.674324 23827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:46.676321 23827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "1b3d371393024dac8b859f1bec7bea35"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "f7d0f1155ac0fb65bd78b63beae417ba"
server_key_iv: "814e72eae6f327aeb929daa7de7ba334"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.676844 23827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "1b3d371393024dac8b859f1bec7bea35"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "f7d0f1155ac0fb65bd78b63beae417ba"
server_key_iv: "814e72eae6f327aeb929daa7de7ba334"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.680431 23827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:20:46.682832 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.684051 23827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:20:46.684166 23827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "1b3d371393024dac8b859f1bec7bea35"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "f7d0f1155ac0fb65bd78b63beae417ba"
server_key_iv: "814e72eae6f327aeb929daa7de7ba334"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.684270 23827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:46.713675 23827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:46.714421 23827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:46.714596 23827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:46.721158 23827 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:37857
I20260928 21:20:46.721175 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:37857 every 8 connection(s)
I20260928 21:20:46.722128 23827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:46.725178 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23827
I20260928 21:20:46.725262 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:46.725297 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:46.725661 5195 external_mini_cluster.cc:1428] Setting key ddfadb3f70ead14f97529c11c0ce3d90
I20260928 21:20:46.731938 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: Bootstrap starting.
I20260928 21:20:46.734476 23894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:46.734879 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.727080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.5.18.254:37857)
0928 21:20:46.727679 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.727693 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.727728 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:20:46.728414 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.728433 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.728823 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.729094 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:46.729432 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.730443 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.731589 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.732096 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.732206 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.733046 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:46.733052 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:46.733064 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:46.733454 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:46.733508 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:46.733528 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:46.733652 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.733774 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.734037 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260928 21:20:46.735317 23894 log.cc:824] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:46.737274 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: No bootstrap required, opened a new log
I20260928 21:20:46.740859 23894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } }
I20260928 21:20:46.741153 23894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:46.741214 23894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3d371393024dac8b859f1bec7bea35, State: Initialized, Role: FOLLOWER
I20260928 21:20:46.741698 23894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } }
I20260928 21:20:46.741835 23894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:46.741899 23894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:46.741993 23894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:46.742920 23894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } }
I20260928 21:20:46.743238 23894 leader_election.cc:302] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b3d371393024dac8b859f1bec7bea35; no voters:
I20260928 21:20:46.743541 23894 leader_election.cc:288] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:46.743649 23899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:46.743870 23899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3d371393024dac8b859f1bec7bea35, State: Running, Role: LEADER
I20260928 21:20:46.744154 23899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } }
I20260928 21:20:46.744742 23894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:46.745708 23900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b3d371393024dac8b859f1bec7bea35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } } }
I20260928 21:20:46.745842 23900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:46.746186 23908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:46.746114 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3d371393024dac8b859f1bec7bea35. Latest consensus state: current_term: 1 leader_uuid: "1b3d371393024dac8b859f1bec7bea35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b3d371393024dac8b859f1bec7bea35" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37857 } } }
I20260928 21:20:46.746799 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:46.749188 23908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:46.754722 23908 catalog_manager.cc:1347] Generated new cluster ID: 30dd8f68c019456db5f7a0467a544382
I20260928 21:20:46.754810 23908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:46.776463 23908 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:46.777486 23908 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:46.786561 23908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: Generated new TSK 0
I20260928 21:20:46.787307 23908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:46.790805 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--builtin_ntp_servers=127.5.18.212:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:46.921180 23918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:46.921463 23918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:46.921501 23918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:46.925625 23918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:46.925709 23918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:46.925791 23918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:46.930696 23918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:46.931819 23918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:46.933251 23918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:46.943241 23926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:46.943243 23924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:46.943228 23923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:46.943610 23918 server_base.cc:1034] running on GCE node
I20260928 21:20:46.944101 23918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:46.944808 23918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:46.945957 23918 hybrid_clock.cc:648] HybridClock initialized: now 1790630446945924 us; error 54 us; skew 500 ppm
I20260928 21:20:46.948078 23918 webserver.cc:460] Webserver started at http://127.5.18.193:39523/ using document root <none> and password file <none>
I20260928 21:20:46.948822 23918 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:46.948896 23918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:46.949128 23918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:46.951560 23918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "9c84089212cc42548a790b3016b9e83c"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "2687a1e339ff48241c57cd7f69c9e922"
server_key_iv: "b80f5bd7e5d53fc5b8e5ac8f7c72a021"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.952098 23918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "9c84089212cc42548a790b3016b9e83c"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "2687a1e339ff48241c57cd7f69c9e922"
server_key_iv: "b80f5bd7e5d53fc5b8e5ac8f7c72a021"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.955716 23918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:20:46.958081 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.959282 23918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:46.959394 23918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "9c84089212cc42548a790b3016b9e83c"
format_stamp: "Formatted at 2026-09-28 21:20:46 on dist-test-slave-rc9f"
server_key: "2687a1e339ff48241c57cd7f69c9e922"
server_key_iv: "b80f5bd7e5d53fc5b8e5ac8f7c72a021"
server_key_version: "encryptionkey@0"
I20260928 21:20:46.959499 23918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:46.977200 23918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:46.977977 23918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:46.978156 23918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:46.978752 23918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:46.979745 23918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:46.979791 23918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.979846 23918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:46.979872 23918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:46.988232 23918 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:44489
I20260928 21:20:46.988256 24045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:44489 every 8 connection(s)
I20260928 21:20:46.989168 23918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:46.997748 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 23918
I20260928 21:20:46.997867 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:46.998150 5195 external_mini_cluster.cc:1428] Setting key 0cad8bc913d5620e367de75543e3c308
I20260928 21:20:46.998638 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.991187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:38265 (local address 127.5.18.254:37857)
0928 21:20:46.991384 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:46.991389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:46.992609 (+ 1220us) server_negotiation.cc:407] Connection header received
0928 21:20:46.994014 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:46.994017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:46.994070 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:46.994166 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:46.995038 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.995585 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.996684 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.996919 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.997010 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.998164 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:46.998168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:46.998171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:46.998186 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:46.998211 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:46.998222 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:46.998298 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:46.998437 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 21:20:46.998506 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 21:20:46.999536 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:46.991723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37857 (local address 127.5.18.193:38265)
0928 21:20:46.992454 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:46.992497 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:46.993805 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:46.994324 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:46.994341 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:46.994429 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:46.994918 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.994930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.995688 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.995690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.996577 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:46.996585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:46.997049 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:46.997052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:46.997202 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:46.997885 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:46.997887 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:46.997942 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:46.998017 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:46.998301 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:46.998310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:46.998315 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:20:46.998418 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:20:46.998720 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":132,"threads_started":1}
I20260928 21:20:47.000145 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--builtin_ntp_servers=127.5.18.212:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:47.000779 24046 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37857
I20260928 21:20:47.001009 24046 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:47.001523 24046 heartbeater.cc:511] Master 127.5.18.254:37857 requested a full tablet report, sending...
I20260928 21:20:47.002938 23858 ts_manager.cc:194] Registered new tserver with Master: 9c84089212cc42548a790b3016b9e83c (127.5.18.193:44489)
I20260928 21:20:47.004148 23858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.193:38265
W20260928 21:20:47.108572 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:47.108850 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:47.108902 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:47.112834 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:47.112896 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:47.112969 24049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:47.117695 24049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:47.118757 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:47.120091 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:47.129580 24054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:47.129581 24057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:47.129585 24055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:47.129894 24049 server_base.cc:1034] running on GCE node
I20260928 21:20:47.130311 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:47.130972 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:47.132130 24049 hybrid_clock.cc:648] HybridClock initialized: now 1790630447132108 us; error 45 us; skew 500 ppm
I20260928 21:20:47.134037 24049 webserver.cc:460] Webserver started at http://127.5.18.194:35837/ using document root <none> and password file <none>
I20260928 21:20:47.134620 24049 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:47.134688 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:47.134886 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:47.136963 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "4a5aa76989494652bbe0c0cf77ace177"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "36114bdb2879ee54b0d9d79054feadc1"
server_key_iv: "8aa47cfeb23b1c4c42db30a98cfa119f"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.137499 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "4a5aa76989494652bbe0c0cf77ace177"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "36114bdb2879ee54b0d9d79054feadc1"
server_key_iv: "8aa47cfeb23b1c4c42db30a98cfa119f"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.141283 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:20:47.143667 24063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.144984 24049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:47.145097 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "4a5aa76989494652bbe0c0cf77ace177"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "36114bdb2879ee54b0d9d79054feadc1"
server_key_iv: "8aa47cfeb23b1c4c42db30a98cfa119f"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.145205 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:47.161551 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:47.162273 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:47.162457 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:47.163026 24049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:47.163986 24049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:47.164031 24049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.164083 24049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:47.164110 24049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.172281 24049 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:38003
I20260928 21:20:47.172302 24176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:38003 every 8 connection(s)
I20260928 21:20:47.173205 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:47.176174 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24049
I20260928 21:20:47.176296 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:47.176620 5195 external_mini_cluster.cc:1428] Setting key 1c3b61f10253c47e9af3fdba7ed487eb
I20260928 21:20:47.178478 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--builtin_ntp_servers=127.5.18.212:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:47.182423 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.174997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:39241 (local address 127.5.18.254:37857)
0928 21:20:47.175144 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.175149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.176362 (+ 1213us) server_negotiation.cc:407] Connection header received
0928 21:20:47.177912 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.177915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.177967 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.178037 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:47.178974 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.179486 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.180601 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.180817 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.180902 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.181946 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:47.181949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:47.181951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:47.181963 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:47.181981 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:47.181989 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:47.182046 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.182183 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.182254 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:20:47.183382 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.175425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37857 (local address 127.5.18.194:39241)
0928 21:20:47.176172 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.176228 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.177712 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.178217 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.178225 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.178312 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:47.178844 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.178859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.179610 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.179613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.180485 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.180493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.180955 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.180959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.181113 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.181694 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:47.181696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:47.181738 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:47.181804 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:47.182056 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:47.182065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:47.182071 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.182176 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.182465 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260928 21:20:47.184479 24177 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37857
I20260928 21:20:47.184752 24177 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:47.185388 24177 heartbeater.cc:511] Master 127.5.18.254:37857 requested a full tablet report, sending...
I20260928 21:20:47.186528 23858 ts_manager.cc:194] Registered new tserver with Master: 4a5aa76989494652bbe0c0cf77ace177 (127.5.18.194:38003)
I20260928 21:20:47.187088 23858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.194:39241
W20260928 21:20:47.288133 24180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:47.288407 24180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:47.288448 24180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:47.292594 24180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:47.292666 24180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:47.292723 24180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:47.297670 24180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:47.298780 24180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:47.300173 24180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:47.309682 24186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:47.309832 24180 server_base.cc:1034] running on GCE node
W20260928 21:20:47.309718 24188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:47.309684 24185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:47.310338 24180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:47.310978 24180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:47.312147 24180 hybrid_clock.cc:648] HybridClock initialized: now 1790630447312125 us; error 38 us; skew 500 ppm
I20260928 21:20:47.314044 24180 webserver.cc:460] Webserver started at http://127.5.18.195:44881/ using document root <none> and password file <none>
I20260928 21:20:47.314617 24180 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:47.314682 24180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:47.314874 24180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:47.316903 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "045321d358f3498ba1270bc72f146487"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "786f5720590b68fc296ca8e0cfa6d03b"
server_key_iv: "07bb143184e3fdd02b044fc93b213e13"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.317459 24180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "045321d358f3498ba1270bc72f146487"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "786f5720590b68fc296ca8e0cfa6d03b"
server_key_iv: "07bb143184e3fdd02b044fc93b213e13"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.321048 24180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:20:47.323459 24194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.324621 24180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:47.324723 24180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "045321d358f3498ba1270bc72f146487"
format_stamp: "Formatted at 2026-09-28 21:20:47 on dist-test-slave-rc9f"
server_key: "786f5720590b68fc296ca8e0cfa6d03b"
server_key_iv: "07bb143184e3fdd02b044fc93b213e13"
server_key_version: "encryptionkey@0"
I20260928 21:20:47.324816 24180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:47.337941 24180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:47.338692 24180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:47.338873 24180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:47.339475 24180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:47.340437 24180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:47.340484 24180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.340538 24180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:47.340565 24180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:47.349030 24180 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:39419
I20260928 21:20:47.349087 24307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:39419 every 8 connection(s)
I20260928 21:20:47.349997 24180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:47.354210 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24180
I20260928 21:20:47.354321 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:47.354637 5195 external_mini_cluster.cc:1428] Setting key 52457d0a732142d6034682cae58cfa11
I20260928 21:20:47.359338 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.351768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:37675 (local address 127.5.18.254:37857)
0928 21:20:47.351909 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.351913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.352991 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 21:20:47.354457 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.354461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.354527 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.354618 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:47.355745 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.356374 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.357609 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.357845 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.357928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.358949 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:47.358952 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:47.358954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:47.358967 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:47.358994 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:47.359003 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:47.359073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.359171 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.359223 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:20:47.360241 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.352185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:37857 (local address 127.5.18.195:37675)
0928 21:20:47.352851 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.352895 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.354216 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.354819 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.354826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.354918 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:47.355595 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.355618 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.356503 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.356507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.357464 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.357474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.357969 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.357972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.358121 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.358693 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:47.358694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:47.358736 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:47.358812 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:47.359069 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:47.359074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:47.359078 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.359166 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.359407 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260928 21:20:47.361340 24308 heartbeater.cc:347] Connected to a master server at 127.5.18.254:37857
I20260928 21:20:47.361557 24308 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:47.362102 24308 heartbeater.cc:511] Master 127.5.18.254:37857 requested a full tablet report, sending...
I20260928 21:20:47.363117 23858 ts_manager.cc:194] Registered new tserver with Master: 045321d358f3498ba1270bc72f146487 (127.5.18.195:39419)
I20260928 21:20:47.363667 23858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.195:37675
I20260928 21:20:47.368709 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:47.374023 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.370589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.5.18.254:37857)
0928 21:20:47.370722 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.370725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.370853 (+ 128us) server_negotiation.cc:407] Connection header received
0928 21:20:47.370945 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.370947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.370984 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.371049 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:47.371254 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.371763 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.372595 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.372823 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.372895 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.373633 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:47.373636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:47.373638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:47.373650 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:47.373668 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:47.373698 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:47.373797 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.373877 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.373908 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:20:47.376861 23858 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:47.379890 23858 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:47.394095 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.388370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:44489 (local address 127.0.0.1:37856)
0928 21:20:47.388798 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.388847 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.388966 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.389700 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.389707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.389732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:47.389879 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.389887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391273 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.391276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.392520 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.392530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.393528 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.393670 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.393724 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.393816 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.393935 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
I20260928 21:20:47.394153 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.388484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.5.18.193:44489)
0928 21:20:47.388968 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.388980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.389005 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:20:47.389102 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.389107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.389333 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.389487 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:47.390012 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391162 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.392651 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.393409 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393524 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.393805 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.393880 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.393989 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":136,"threads_started":1}
I20260928 21:20:47.394622 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.388449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39419 (local address 127.0.0.1:42972)
0928 21:20:47.389791 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.389803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.389943 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.390418 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.390420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.390434 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:47.390526 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.390532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391669 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.391672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.392848 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.392858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.394235 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.394239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.394349 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.394373 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.394443 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.394510 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260928 21:20:47.395020 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.389174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.5.18.195:39419)
0928 21:20:47.389656 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.389668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.389877 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:20:47.390088 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.390093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.390292 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.390439 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:47.390624 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391566 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.392973 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.393879 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393991 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.394151 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.394788 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.394894 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":155,"threads_started":1}
I20260928 21:20:47.397187 23980 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae687a392984b53af86bb728c0eaecf (DEFAULT_TABLE table=test-table [id=1ce0b3c8c46142a1a7d47bd1a738bf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:47.397336 24242 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae687a392984b53af86bb728c0eaecf (DEFAULT_TABLE table=test-table [id=1ce0b3c8c46142a1a7d47bd1a738bf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:47.397742 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.389120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38003 (local address 127.0.0.1:33448)
0928 21:20:47.390109 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.390119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.390192 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.390638 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.390641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.390655 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:47.390724 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.390730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391837 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.391840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.392940 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.392947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.393972 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.394057 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.394077 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.394140 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.394202 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":85,"threads_started":1}
I20260928 21:20:47.398263 24242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae687a392984b53af86bb728c0eaecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:47.398262 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.389513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.5.18.194:38003)
0928 21:20:47.389935 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.389947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.390209 (+ 262us) server_negotiation.cc:407] Connection header received
0928 21:20:47.390297 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.390302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.390464 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.390608 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:47.390817 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.391724 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393040 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.393649 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.393742 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.393915 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.397965 (+ 4050us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.398087 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":157,"threads_started":1}
I20260928 21:20:47.398270 23980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae687a392984b53af86bb728c0eaecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:47.401021 24111 tablet_service.cc:1467] Processing CreateTablet for tablet 9ae687a392984b53af86bb728c0eaecf (DEFAULT_TABLE table=test-table [id=1ce0b3c8c46142a1a7d47bd1a738bf3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:47.401912 24111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ae687a392984b53af86bb728c0eaecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:47.405555 24325 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Bootstrap starting.
I20260928 21:20:47.407406 24325 tablet_bootstrap.cc:654] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:47.408167 24325 log.cc:824] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:47.408214 24326 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Bootstrap starting.
I20260928 21:20:47.409874 24328 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Bootstrap starting.
I20260928 21:20:47.410234 24325 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: No bootstrap required, opened a new log
I20260928 21:20:47.410415 24325 ts_tablet_manager.cc:1397] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:20:47.410866 24326 tablet_bootstrap.cc:654] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:47.411824 24326 log.cc:824] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:47.411922 24328 tablet_bootstrap.cc:654] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:47.412642 24328 log.cc:824] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:47.413796 24326 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: No bootstrap required, opened a new log
I20260928 21:20:47.413990 24326 ts_tablet_manager.cc:1397] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 21:20:47.414233 24328 tablet_bootstrap.cc:492] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: No bootstrap required, opened a new log
I20260928 21:20:47.414412 24328 ts_tablet_manager.cc:1397] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:20:47.414491 24325 raft_consensus.cc:348] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.414774 24325 raft_consensus.cc:374] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:47.414831 24325 raft_consensus.cc:729] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045321d358f3498ba1270bc72f146487, State: Initialized, Role: FOLLOWER
I20260928 21:20:47.415273 24325 consensus_queue.cc:260] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.415903 24308 heartbeater.cc:503] Master 127.5.18.254:37857 was elected leader, sending a full tablet report...
I20260928 21:20:47.417295 24325 ts_tablet_manager.cc:1428] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:20:47.417869 24328 raft_consensus.cc:348] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.418116 24328 raft_consensus.cc:374] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:47.418181 24328 raft_consensus.cc:729] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c84089212cc42548a790b3016b9e83c, State: Initialized, Role: FOLLOWER
I20260928 21:20:47.418381 24326 raft_consensus.cc:348] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.418596 24326 raft_consensus.cc:374] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:47.418643 24326 raft_consensus.cc:729] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5aa76989494652bbe0c0cf77ace177, State: Initialized, Role: FOLLOWER
I20260928 21:20:47.418673 24328 consensus_queue.cc:260] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.419065 24326 consensus_queue.cc:260] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.419553 24328 ts_tablet_manager.cc:1428] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:20:47.419750 24177 heartbeater.cc:503] Master 127.5.18.254:37857 was elected leader, sending a full tablet report...
I20260928 21:20:47.419863 24046 heartbeater.cc:503] Master 127.5.18.254:37857 was elected leader, sending a full tablet report...
I20260928 21:20:47.419970 24326 ts_tablet_manager.cc:1428] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
W20260928 21:20:47.424049 24178 tablet.cc:2369] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:20:47.490412 24047 tablet.cc:2369] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:47.533524 24333 raft_consensus.cc:482] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:47.533771 24333 raft_consensus.cc:504] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.534991 24333 leader_election.cc:288] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c84089212cc42548a790b3016b9e83c (127.5.18.193:44489), 045321d358f3498ba1270bc72f146487 (127.5.18.195:39419)
I20260928 21:20:47.538808 24323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.535381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:45367 (local address 127.5.18.195:39419)
0928 21:20:47.535510 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.535514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.535531 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:20:47.535620 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.535623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.535675 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.535759 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:47.536059 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.536685 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.537854 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.538273 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.538368 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.538420 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.538601 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.538676 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:20:47.538818 24321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.535382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:60665 (local address 127.5.18.193:44489)
0928 21:20:47.535510 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.535514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.535844 (+ 330us) server_negotiation.cc:407] Connection header received
0928 21:20:47.535967 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.535970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.536025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.536091 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:47.536346 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.536882 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.537926 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.538308 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.538417 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.538465 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.538635 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.538717 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:20:47.538816 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.535271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:39419 (local address 127.5.18.194:45367)
0928 21:20:47.535406 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.535421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.535521 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.535780 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.535782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.535800 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:47.535945 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.535952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.536790 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.536794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.537743 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.537753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.538391 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.538394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.538490 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.538511 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.538570 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.538642 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":26,"spinlock_wait_cycles":18688}
I20260928 21:20:47.539060 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.535273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:44489 (local address 127.5.18.194:60665)
0928 21:20:47.535745 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.535758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:47.535871 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:47.536114 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:47.536117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:47.536135 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:47.536246 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.536253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.536984 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.536986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.537811 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:47.537822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.538431 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.538435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:47.538524 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.538541 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:20:47.538609 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 21:20:47.538677 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":140,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260928 21:20:47.539541 24262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae687a392984b53af86bb728c0eaecf" candidate_uuid: "4a5aa76989494652bbe0c0cf77ace177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045321d358f3498ba1270bc72f146487" is_pre_election: true
I20260928 21:20:47.539726 24000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae687a392984b53af86bb728c0eaecf" candidate_uuid: "4a5aa76989494652bbe0c0cf77ace177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c84089212cc42548a790b3016b9e83c" is_pre_election: true
I20260928 21:20:47.539898 24262 raft_consensus.cc:2454] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5aa76989494652bbe0c0cf77ace177 in term 0.
I20260928 21:20:47.540023 24000 raft_consensus.cc:2454] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5aa76989494652bbe0c0cf77ace177 in term 0.
I20260928 21:20:47.540354 24066 leader_election.cc:302] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045321d358f3498ba1270bc72f146487, 4a5aa76989494652bbe0c0cf77ace177; no voters:
I20260928 21:20:47.540620 24333 raft_consensus.cc:2781] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:47.540709 24333 raft_consensus.cc:482] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:47.540740 24333 raft_consensus.cc:3037] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:47.541765 24333 raft_consensus.cc:504] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.542107 24333 leader_election.cc:288] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [CANDIDATE]: Term 1 election: Requested vote from peers 9c84089212cc42548a790b3016b9e83c (127.5.18.193:44489), 045321d358f3498ba1270bc72f146487 (127.5.18.195:39419)
I20260928 21:20:47.542439 24000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae687a392984b53af86bb728c0eaecf" candidate_uuid: "4a5aa76989494652bbe0c0cf77ace177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c84089212cc42548a790b3016b9e83c"
I20260928 21:20:47.542439 24262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ae687a392984b53af86bb728c0eaecf" candidate_uuid: "4a5aa76989494652bbe0c0cf77ace177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045321d358f3498ba1270bc72f146487"
I20260928 21:20:47.542564 24000 raft_consensus.cc:3037] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:47.542564 24262 raft_consensus.cc:3037] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:47.543780 24262 raft_consensus.cc:2454] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5aa76989494652bbe0c0cf77ace177 in term 1.
I20260928 21:20:47.543780 24000 raft_consensus.cc:2454] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a5aa76989494652bbe0c0cf77ace177 in term 1.
I20260928 21:20:47.544070 24067 leader_election.cc:302] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5aa76989494652bbe0c0cf77ace177, 9c84089212cc42548a790b3016b9e83c; no voters:
I20260928 21:20:47.544269 24333 raft_consensus.cc:2781] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:47.544473 24333 raft_consensus.cc:686] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 1 LEADER]: Becoming Leader. State: Replica: 4a5aa76989494652bbe0c0cf77ace177, State: Running, Role: LEADER
I20260928 21:20:47.544816 24333 consensus_queue.cc:237] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } }
I20260928 21:20:47.548044 23857 catalog_manager.cc:5495] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a5aa76989494652bbe0c0cf77ace177 (127.5.18.194). New cstate: current_term: 1 leader_uuid: "4a5aa76989494652bbe0c0cf77ace177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a5aa76989494652bbe0c0cf77ace177" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38003 } health_report { overall_health: HEALTHY } } }
I20260928 21:20:47.584980 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.581748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.5.18.194:38003)
0928 21:20:47.581875 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:47.581879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:47.581894 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:20:47.581968 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:47.581971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:47.582027 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:47.582115 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:47.582362 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.582857 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.583643 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:47.583858 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:47.583933 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:47.584222 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:47.584637 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:47.584747 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:47.584801 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:20:47.584833 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:20:47.591070 24262 raft_consensus.cc:1261] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 1 FOLLOWER]: Refusing update from remote peer 4a5aa76989494652bbe0c0cf77ace177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:47.591109 24000 raft_consensus.cc:1261] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 1 FOLLOWER]: Refusing update from remote peer 4a5aa76989494652bbe0c0cf77ace177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:47.591751 24333 consensus_queue.cc:1035] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c84089212cc42548a790b3016b9e83c" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:47.591990 24335 consensus_queue.cc:1035] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "045321d358f3498ba1270bc72f146487" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 39419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:47.598439 24338 mvcc.cc:205] Tried to move back new op lower bound from 7334422313325731840 to 7334422313146273792. Current Snapshot: MvccSnapshot[applied={T|T < 7334422313325731840}]
I20260928 21:20:47.601541 24339 mvcc.cc:205] Tried to move back new op lower bound from 7334422313325731840 to 7334422313146273792. Current Snapshot: MvccSnapshot[applied={T|T < 7334422313325731840}]
I20260928 21:20:47.610606 23855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:47.610817 23855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:47.613538 23855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b3d371393024dac8b859f1bec7bea35: Sending DeleteTablet for 3 replicas of tablet 9ae687a392984b53af86bb728c0eaecf
I20260928 21:20:47.614512 24242 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae687a392984b53af86bb728c0eaecf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:47 UTC) from {username='slave'} at 127.0.0.1:42972
I20260928 21:20:47.614684 24111 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae687a392984b53af86bb728c0eaecf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:47 UTC) from {username='slave'} at 127.0.0.1:33448
I20260928 21:20:47.614828 23980 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ae687a392984b53af86bb728c0eaecf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:47 UTC) from {username='slave'} at 127.0.0.1:37856
I20260928 21:20:47.615128 24347 tablet_replica.cc:330] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: stopping tablet replica
I20260928 21:20:47.615340 24349 tablet_replica.cc:330] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c: stopping tablet replica
I20260928 21:20:47.615583 24347 raft_consensus.cc:2229] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:47.615588 24349 raft_consensus.cc:2229] T 9ae687a392984b53af86bb728c0eaecf P 9c84089212cc42548a790b3016b9e83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:47.615649 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23918
I20260928 21:20:47.616997 24347 raft_consensus.cc:2258] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:47.617511 24348 tablet_replica.cc:330] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: stopping tablet replica
I20260928 21:20:47.617915 24348 raft_consensus.cc:2229] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:47.618269 24348 raft_consensus.cc:2258] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:47.618757 24347 ts_tablet_manager.cc:1905] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:47.619568 24348 ts_tablet_manager.cc:1905] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:47.621383 24347 ts_tablet_manager.cc:1918] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:47.621487 24347 log.cc:1196] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/9ae687a392984b53af86bb728c0eaecf
I20260928 21:20:47.621814 24347 ts_tablet_manager.cc:1939] T 9ae687a392984b53af86bb728c0eaecf P 045321d358f3498ba1270bc72f146487: Deleting consensus metadata
I20260928 21:20:47.622758 24348 ts_tablet_manager.cc:1918] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:47.622934 24348 log.cc:1196] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/9ae687a392984b53af86bb728c0eaecf
I20260928 21:20:47.623190 23844 catalog_manager.cc:4841] TS 045321d358f3498ba1270bc72f146487 (127.5.18.195:39419): tablet 9ae687a392984b53af86bb728c0eaecf (table test-table [id=1ce0b3c8c46142a1a7d47bd1a738bf3b]) successfully deleted
I20260928 21:20:47.623368 24348 ts_tablet_manager.cc:1939] T 9ae687a392984b53af86bb728c0eaecf P 4a5aa76989494652bbe0c0cf77ace177: Deleting consensus metadata
W20260928 21:20:47.623793 23845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:44489 (error 108)
I20260928 21:20:47.624660 23845 catalog_manager.cc:4841] TS 4a5aa76989494652bbe0c0cf77ace177 (127.5.18.194:38003): tablet 9ae687a392984b53af86bb728c0eaecf (table test-table [id=1ce0b3c8c46142a1a7d47bd1a738bf3b]) successfully deleted
I20260928 21:20:47.624815 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:47.624517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:44489 (local address 127.0.0.1:37866)
0928 21:20:47.624642 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:47.624727 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:20:47.624940 23845 catalog_manager.cc:4584] TS 9c84089212cc42548a790b3016b9e83c (127.5.18.193:44489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ae687a392984b53af86bb728c0eaecf: Network error: Client connection negotiation failed: client connection to 127.5.18.193:44489: connect: Connection refused (error 111)
I20260928 21:20:47.625275 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24049
I20260928 21:20:47.630405 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24180
I20260928 21:20:47.635106 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 23827
2026-09-28T21:20:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1122 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:47 dist-test-slave-rc9f krb5kdc[24353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:47 dist-test-slave-rc9f krb5kdc[24353](info): set up 2 sockets
Sep 28 21:20:47 dist-test-slave-rc9f krb5kdc[24353](info): commencing operation
krb5kdc: starting...
W20260928 21:20:48.597319 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:48 dist-test-slave-rc9f krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:20:48.716588 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44991
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:48.824005 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:48.824406 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:48.824473 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:48.828663 24366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:48.828742 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:48.828775 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:48.828795 24366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:48.828822 24366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:48.834336 24366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:44991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:48.835443 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:48.836468 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:48.845024 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:48.845019 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:48.845013 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:48.845105 24366 server_base.cc:1034] running on GCE node
I20260928 21:20:48.845985 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:48.847178 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:48.848379 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790630448848355 us; error 45 us; skew 500 ppm
Sep 28 21:20:48 dist-test-slave-rc9f krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630448, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:48.851332 24366 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:20:48.852380 24366 webserver.cc:460] Webserver started at http://127.5.18.254:36709/ using document root <none> and password file <none>
I20260928 21:20:48.852980 24366 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:48.853048 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:48.853255 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:48.855546 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "205eda36e9704747b2cc5d88b6ee015d"
format_stamp: "Formatted at 2026-09-28 21:20:48 on dist-test-slave-rc9f"
server_key: "a7f806e3b38d29b9574173d42f7d431b"
server_key_iv: "0ab105d144abe70b587f95b892656f09"
server_key_version: "encryptionkey@0"
I20260928 21:20:48.856155 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "205eda36e9704747b2cc5d88b6ee015d"
format_stamp: "Formatted at 2026-09-28 21:20:48 on dist-test-slave-rc9f"
server_key: "a7f806e3b38d29b9574173d42f7d431b"
server_key_iv: "0ab105d144abe70b587f95b892656f09"
server_key_version: "encryptionkey@0"
I20260928 21:20:48.860325 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:20:48.863116 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:48.864516 24366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:20:48.864660 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "205eda36e9704747b2cc5d88b6ee015d"
format_stamp: "Formatted at 2026-09-28 21:20:48 on dist-test-slave-rc9f"
server_key: "a7f806e3b38d29b9574173d42f7d431b"
server_key_iv: "0ab105d144abe70b587f95b892656f09"
server_key_version: "encryptionkey@0"
I20260928 21:20:48.864789 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:48.899179 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:48.902401 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:48.902601 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:48.910012 24366 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:44991
I20260928 21:20:48.910039 24433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:44991 every 8 connection(s)
I20260928 21:20:48.911055 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:48.913074 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24366
I20260928 21:20:48.913218 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:48.913535 5195 external_mini_cluster.cc:1428] Setting key 8dd22cc999a703937d6b59fe05576931
I20260928 21:20:48.914368 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:48.920449 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: Bootstrap starting.
Sep 28 21:20:48 dist-test-slave-rc9f krb5kdc[24353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:48.923200 24434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:48.924106 24434 log.cc:824] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:48.926364 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: No bootstrap required, opened a new log
I20260928 21:20:48.927873 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:48.915124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.5.18.254:44991)
0928 21:20:48.915594 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:48.915611 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:48.915649 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:20:48.916326 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:48.916344 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:48.916714 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:48.917035 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:48.917886 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:48.918895 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:48.919749 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:48.920253 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:48.920402 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:48.923135 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:48.923141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:48.923156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:48.923204 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:48.924986 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:48.925432 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:48.925438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:48.925443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:48.925557 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:48.925886 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:48.925889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:48.925891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:48.926326 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:48.926515 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:48.926798 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 21:20:48.927105 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":159,"threads_started":1}
I20260928 21:20:48.930534 24434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } }
I20260928 21:20:48.930807 24434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:48.930867 24434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205eda36e9704747b2cc5d88b6ee015d, State: Initialized, Role: FOLLOWER
I20260928 21:20:48.931313 24434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } }
I20260928 21:20:48.931447 24434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:48.931511 24434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:48.931605 24434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:48.932574 24434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } }
I20260928 21:20:48.932912 24434 leader_election.cc:302] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 205eda36e9704747b2cc5d88b6ee015d; no voters:
I20260928 21:20:48.933239 24434 leader_election.cc:288] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:48.933338 24439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:48.933601 24439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [term 1 LEADER]: Becoming Leader. State: Replica: 205eda36e9704747b2cc5d88b6ee015d, State: Running, Role: LEADER
I20260928 21:20:48.933935 24439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } }
I20260928 21:20:48.934546 24434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:48.935655 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "205eda36e9704747b2cc5d88b6ee015d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } } }
I20260928 21:20:48.935808 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:48.936127 24441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205eda36e9704747b2cc5d88b6ee015d. Latest consensus state: current_term: 1 leader_uuid: "205eda36e9704747b2cc5d88b6ee015d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205eda36e9704747b2cc5d88b6ee015d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44991 } } }
I20260928 21:20:48.936192 24448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:48.936223 24441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:48.939355 24448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:48.947093 24448 catalog_manager.cc:1347] Generated new cluster ID: 0532daded2664cf6b6bbbbbddb0b4d98
I20260928 21:20:48.947187 24448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:48.972252 24448 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:48.973243 24448 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:48.982173 24448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: Generated new TSK 0
I20260928 21:20:48.982944 24448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.18.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
Entry for principal kudu/127.5.18.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab.
I20260928 21:20:49.017352 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:49.133755 24460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:49.134048 24460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:49.134093 24460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:49.138497 24460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:49.138572 24460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:49.138638 24460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:49.144209 24460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.193.keytab
--principal=kudu/127.5.18.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:49.145331 24460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:49.146390 24460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:49.156595 24465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:49.156688 24460 server_base.cc:1034] running on GCE node
W20260928 21:20:49.156595 24468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:49.156595 24466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:49.157336 24460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:49.158084 24460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:49.159258 24460 hybrid_clock.cc:648] HybridClock initialized: now 1790630449159232 us; error 44 us; skew 500 ppm
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:49.162147 24460 init.cc:377] Logged in from keytab as kudu/127.5.18.193@KRBTEST.COM (short username kudu)
I20260928 21:20:49.163197 24460 webserver.cc:460] Webserver started at http://127.5.18.193:36489/ using document root <none> and password file <none>
I20260928 21:20:49.163760 24460 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:49.163828 24460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:49.164023 24460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:49.166157 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "0202adbcb9954b3e945bec70aba62459"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "a2f4101ba7c8d3774f56590722241233"
server_key_iv: "99a395f6e3a1b18f6db6ca5a350e72e6"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.166705 24460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "0202adbcb9954b3e945bec70aba62459"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "a2f4101ba7c8d3774f56590722241233"
server_key_iv: "99a395f6e3a1b18f6db6ca5a350e72e6"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.170575 24460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:49.173046 24475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.174439 24460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:49.174561 24460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "0202adbcb9954b3e945bec70aba62459"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "a2f4101ba7c8d3774f56590722241233"
server_key_iv: "99a395f6e3a1b18f6db6ca5a350e72e6"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.174674 24460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:49.193181 24460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:49.196547 24460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:49.196785 24460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:49.197430 24460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:49.198388 24460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:49.198439 24460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.198495 24460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:49.198523 24460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.208163 24460 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:34083
I20260928 21:20:49.208226 24588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:34083 every 8 connection(s)
I20260928 21:20:49.209183 24460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:49.214080 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24460
I20260928 21:20:49.214234 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:49.214534 5195 external_mini_cluster.cc:1428] Setting key 88de3a318de2f95d657c732d080e3819
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.193@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.194@KRBTEST.COM; defaulting to no policy
I20260928 21:20:49.224196 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.211242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:48039 (local address 127.5.18.254:44991)
0928 21:20:49.211409 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.211413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.212311 (+ 898us) server_negotiation.cc:407] Connection header received
0928 21:20:49.213194 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.213197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.213251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.213337 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:49.214717 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.215381 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.216756 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.216986 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.217076 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.219968 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:49.219973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:49.219976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:49.220013 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:49.221798 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.222382 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.222386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.222388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.222494 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.222920 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.222923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.222925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.223173 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:49.223281 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.223869 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.224012 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:20:49.225023 24591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.211621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:44991 (local address 127.5.18.193:48039)
0928 21:20:49.212125 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.212192 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.212988 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.213504 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.213512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.213929 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:49.214533 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.214551 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.215522 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.215525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.216591 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.216602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.217142 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.217156 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.217329 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.217967 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:49.217985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:49.219742 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:49.221939 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.221945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.221960 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.222275 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.222622 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.222625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.222626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.222810 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.223303 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:49.223310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:49.223588 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.223786 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.224080 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":131,"threads_started":1}
I20260928 21:20:49.226493 24589 heartbeater.cc:347] Connected to a master server at 127.5.18.254:44991
I20260928 21:20:49.226831 24589 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.194@KRBTEST.COM" created.
I20260928 21:20:49.227384 24589 heartbeater.cc:511] Master 127.5.18.254:44991 requested a full tablet report, sending...
I20260928 21:20:49.229043 24398 ts_manager.cc:194] Registered new tserver with Master: 0202adbcb9954b3e945bec70aba62459 (127.5.18.193:34083)
I20260928 21:20:49.230357 24398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.193@KRBTEST.COM'} at 127.5.18.193:48039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
Entry for principal kudu/127.5.18.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab.
I20260928 21:20:49.244609 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:49.360399 24594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:49.360697 24594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:49.360760 24594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:49.365010 24594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:49.365087 24594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:49.365171 24594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:49.370822 24594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.194.keytab
--principal=kudu/127.5.18.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:49.371976 24594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:49.373061 24594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:49.383067 24599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:49.383033 24602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:49.383206 24594 server_base.cc:1034] running on GCE node
W20260928 21:20:49.383400 24600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:49.383811 24594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:49.384474 24594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:49.385664 24594 hybrid_clock.cc:648] HybridClock initialized: now 1790630449385607 us; error 42 us; skew 500 ppm
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:49.388258 24594 init.cc:377] Logged in from keytab as kudu/127.5.18.194@KRBTEST.COM (short username kudu)
I20260928 21:20:49.389107 24594 webserver.cc:460] Webserver started at http://127.5.18.194:41283/ using document root <none> and password file <none>
I20260928 21:20:49.389672 24594 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:49.389735 24594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:49.389912 24594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:49.391934 24594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "70a0c553f923408a874dc948e49f7c09"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "5eb757cdd7f89ab00786037fbe13d9ff"
server_key_iv: "db0bf3ac203891229534b4ab803f4e23"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.392460 24594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "70a0c553f923408a874dc948e49f7c09"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "5eb757cdd7f89ab00786037fbe13d9ff"
server_key_iv: "db0bf3ac203891229534b4ab803f4e23"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.396412 24594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:20:49.399240 24609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.400619 24594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:20:49.400771 24594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "70a0c553f923408a874dc948e49f7c09"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "5eb757cdd7f89ab00786037fbe13d9ff"
server_key_iv: "db0bf3ac203891229534b4ab803f4e23"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.400895 24594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:49.414925 24594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:49.418375 24594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:49.418573 24594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:49.419206 24594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:49.420109 24594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:49.420158 24594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.420213 24594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:49.420241 24594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.429347 24594 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:38805
I20260928 21:20:49.429388 24722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:38805 every 8 connection(s)
I20260928 21:20:49.430591 24594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:49.431370 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24594
I20260928 21:20:49.431509 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:49.431897 5195 external_mini_cluster.cc:1428] Setting key 749d7de7fdd2b09a2dac29559439f3d5
WARNING: no policy specified for kudu/127.5.18.195@KRBTEST.COM; defaulting to no policy
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.194@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.195@KRBTEST.COM" created.
I20260928 21:20:49.444914 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.432714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:44443 (local address 127.5.18.254:44991)
0928 21:20:49.432847 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.432851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.433981 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 21:20:49.435068 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.435071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.435131 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.435219 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:49.436784 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.437319 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.438547 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.438771 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.438867 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.441507 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:49.441510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:49.441511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:49.441538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:49.442840 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.443339 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.443343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.443345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.443423 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.443737 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.443740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.443742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.443978 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:49.444065 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.444598 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.444736 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:49.445837 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.433157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:44991 (local address 127.5.18.194:44443)
0928 21:20:49.433754 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.433855 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.434858 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.435539 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.435549 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.435964 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:49.436617 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.436655 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.437443 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.437448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.438427 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.438437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.438927 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.438931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.439111 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.439677 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:49.439695 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:49.441265 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:49.442953 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.442960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.442979 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.443243 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.443503 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.443506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.443508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.443651 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.444078 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:49.444086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:49.444362 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.444540 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.444901 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
I20260928 21:20:49.447036 24723 heartbeater.cc:347] Connected to a master server at 127.5.18.254:44991
I20260928 21:20:49.447288 24723 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:49.447821 24723 heartbeater.cc:511] Master 127.5.18.254:44991 requested a full tablet report, sending...
I20260928 21:20:49.448956 24398 ts_manager.cc:194] Registered new tserver with Master: 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805)
I20260928 21:20:49.449792 24398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.194@KRBTEST.COM'} at 127.5.18.194:44443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
Entry for principal kudu/127.5.18.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab.
I20260928 21:20:49.460986 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:49.573383 24728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:49.573722 24728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:49.573769 24728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:49.578102 24728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:49.578197 24728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:49.578260 24728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:49.584035 24728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.195.keytab
--principal=kudu/127.5.18.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:44991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:49.585125 24728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:49.586187 24728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:49.596164 24734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:49.596158 24736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:49.596159 24733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:49.596495 24728 server_base.cc:1034] running on GCE node
I20260928 21:20:49.596953 24728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:49.597687 24728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:49.598861 24728 hybrid_clock.cc:648] HybridClock initialized: now 1790630449598841 us; error 38 us; skew 500 ppm
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:49.601922 24728 init.cc:377] Logged in from keytab as kudu/127.5.18.195@KRBTEST.COM (short username kudu)
I20260928 21:20:49.602936 24728 webserver.cc:460] Webserver started at http://127.5.18.195:45311/ using document root <none> and password file <none>
I20260928 21:20:49.603520 24728 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:49.603592 24728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:49.603803 24728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:49.606184 24728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "c40b6ec1cea24808abd43d63001958f5"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "28807f6a392038e1c64276595195f579"
server_key_iv: "0834744c930f99537d92fb723c2e2f9e"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.606779 24728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "c40b6ec1cea24808abd43d63001958f5"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "28807f6a392038e1c64276595195f579"
server_key_iv: "0834744c930f99537d92fb723c2e2f9e"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.610734 24728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:49.613399 24743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.614766 24728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:49.614887 24728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "c40b6ec1cea24808abd43d63001958f5"
format_stamp: "Formatted at 2026-09-28 21:20:49 on dist-test-slave-rc9f"
server_key: "28807f6a392038e1c64276595195f579"
server_key_iv: "0834744c930f99537d92fb723c2e2f9e"
server_key_version: "encryptionkey@0"
I20260928 21:20:49.614995 24728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:49.626221 24728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:49.629403 24728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:49.629710 24728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:49.630393 24728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:49.631382 24728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:49.631431 24728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.631487 24728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:49.631515 24728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:49.640523 24728 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:43509
I20260928 21:20:49.640543 24856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:43509 every 8 connection(s)
I20260928 21:20:49.641494 24728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:49.647495 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24728
I20260928 21:20:49.647606 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:49.647929 5195 external_mini_cluster.cc:1428] Setting key 02aa5540130a12cbec685c737bbfdf53
Sep 28 21:20:49 dist-test-slave-rc9f krb5kdc[24353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.195@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:20:49.656450 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.643486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:45791 (local address 127.5.18.254:44991)
0928 21:20:49.643639 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.643643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.644561 (+ 918us) server_negotiation.cc:407] Connection header received
0928 21:20:49.645959 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.645963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.646030 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.646102 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:49.647466 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.648078 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.649543 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.649957 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.650066 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.652645 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:49.652649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:49.652651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:49.652681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:49.654064 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.654786 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.654790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.654791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.654885 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.655234 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.655238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.655239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.655499 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:49.655589 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.656189 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.656315 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:20:49.657255 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.643869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:44991 (local address 127.5.18.195:45791)
0928 21:20:49.644379 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.644429 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.645718 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.646315 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.646323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.646734 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:49.647301 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.647317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.648216 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.648218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.649396 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.649408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.650127 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.650132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.650337 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.650900 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:20:49.650916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:49.652446 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:49.654198 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.654205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.654220 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.654587 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.654966 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:20:49.654968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:20:49.654970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:20:49.655124 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:20:49.655630 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:49.655636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:49.655926 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.656099 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.656391 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260928 21:20:49.658463 24857 heartbeater.cc:347] Connected to a master server at 127.5.18.254:44991
I20260928 21:20:49.658728 24857 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:49.659256 24857 heartbeater.cc:511] Master 127.5.18.254:44991 requested a full tablet report, sending...
I20260928 21:20:49.660421 24398 ts_manager.cc:194] Registered new tserver with Master: c40b6ec1cea24808abd43d63001958f5 (127.5.18.195:43509)
I20260928 21:20:49.661077 24398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.18.195@KRBTEST.COM'} at 127.5.18.195:45791
I20260928 21:20:49.662248 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:49.671809 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.664277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.5.18.254:44991)
0928 21:20:49.664412 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.664417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.664598 (+ 181us) server_negotiation.cc:407] Connection header received
0928 21:20:49.664788 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.664790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.664836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.664911 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:49.665666 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.666198 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.667096 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.667326 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.667413 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.668905 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:49.668908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:49.668910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:49.668936 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:49.670249 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.670627 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.670630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.670631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.670710 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:49.670939 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:49.670942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:49.670943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:49.671198 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:49.671307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.671508 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.671628 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:20:49.675058 24398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:49.678272 24398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:49.696204 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.689947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38805 (local address 127.0.0.1:33764)
0928 21:20:49.691005 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.691051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.691190 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.691951 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.691957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.691982 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:49.692143 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.692150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.693256 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.693260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.694608 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.694617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.695624 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.695627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.695746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.695793 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.695890 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.696019 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":19,"spinlock_wait_cycles":4736,"thread_start_us":127,"threads_started":1}
I20260928 21:20:49.696272 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.689829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.5.18.194:38805)
0928 21:20:49.691221 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.691235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.691260 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:20:49.691337 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.691342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.691572 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.691720 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:49.692276 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.693110 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.694739 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.695515 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.695737 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.695936 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.695999 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.696108 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282,"thread_start_us":130,"threads_started":1}
I20260928 21:20:49.697901 24872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.691043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:43509 (local address 127.0.0.1:45718)
0928 21:20:49.691738 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.691752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.691851 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.693272 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.693278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.693302 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:49.693427 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.693434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.695012 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.695015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.696049 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.696057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.697410 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.697413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.697520 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.697631 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.697695 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.697778 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":56,"thread_start_us":111,"threads_started":1}
I20260928 21:20:49.698153 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.689781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34083 (local address 127.0.0.1:34648)
0928 21:20:49.690977 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:49.691046 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:49.691190 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:49.692143 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:49.692146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:49.692161 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:49.692247 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.692264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.694365 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.694369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.695468 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:49.695478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.697529 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.697532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:49.697823 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.697851 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:20:49.697939 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:20:49.698014 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":152,"spinlock_wait_cycles":4352,"thread_start_us":88,"threads_started":1}
I20260928 21:20:49.698776 24873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.691764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.5.18.195:43509)
0928 21:20:49.692109 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.692120 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.692616 (+ 496us) server_negotiation.cc:407] Connection header received
0928 21:20:49.692786 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.692792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.692977 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.693612 (+ 635us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:49.693769 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.694792 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.696182 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.697280 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.698185 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.698458 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.698516 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.698623 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":143,"threads_started":1}
I20260928 21:20:49.698774 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:49.690616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.5.18.193:34083)
0928 21:20:49.691467 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:49.691481 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:49.691501 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:20:49.691808 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:49.691814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:49.692004 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:49.692154 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:49.692857 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.694208 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.695607 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:49.697182 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:49.697305 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:49.697476 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:49.698194 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 21:20:49.698626 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":122,"threads_started":1}
I20260928 21:20:49.700405 24657 tablet_service.cc:1467] Processing CreateTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 (DEFAULT_TABLE table=test-table [id=1a2930381b2a4fcb933970b57db9e0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:49.701985 24657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d4285d62cd4cb5bd7619eb3bb77390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:49.702374 24791 tablet_service.cc:1467] Processing CreateTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 (DEFAULT_TABLE table=test-table [id=1a2930381b2a4fcb933970b57db9e0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:49.702394 24523 tablet_service.cc:1467] Processing CreateTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 (DEFAULT_TABLE table=test-table [id=1a2930381b2a4fcb933970b57db9e0ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:49.703344 24791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d4285d62cd4cb5bd7619eb3bb77390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:49.703418 24523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8d4285d62cd4cb5bd7619eb3bb77390. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:49.709753 24874 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Bootstrap starting.
I20260928 21:20:49.711069 24876 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Bootstrap starting.
I20260928 21:20:49.712221 24874 tablet_bootstrap.cc:654] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:49.713177 24876 tablet_bootstrap.cc:654] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:49.713495 24874 log.cc:824] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:49.714089 24875 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Bootstrap starting.
I20260928 21:20:49.714324 24876 log.cc:824] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:49.715914 24874 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: No bootstrap required, opened a new log
I20260928 21:20:49.716073 24876 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: No bootstrap required, opened a new log
I20260928 21:20:49.716136 24874 ts_tablet_manager.cc:1397] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 21:20:49.716281 24876 ts_tablet_manager.cc:1397] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:20:49.716496 24875 tablet_bootstrap.cc:654] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:49.717955 24875 log.cc:824] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:49.720041 24874 raft_consensus.cc:348] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.720369 24874 raft_consensus.cc:374] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:49.720168 24876 raft_consensus.cc:348] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.720422 24874 raft_consensus.cc:729] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40b6ec1cea24808abd43d63001958f5, State: Initialized, Role: FOLLOWER
I20260928 21:20:49.720484 24876 raft_consensus.cc:374] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:49.720544 24876 raft_consensus.cc:729] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0202adbcb9954b3e945bec70aba62459, State: Initialized, Role: FOLLOWER
I20260928 21:20:49.720901 24875 tablet_bootstrap.cc:492] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: No bootstrap required, opened a new log
I20260928 21:20:49.720935 24874 consensus_queue.cc:260] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.721036 24876 consensus_queue.cc:260] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.721131 24875 ts_tablet_manager.cc:1397] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:20:49.721721 24857 heartbeater.cc:503] Master 127.5.18.254:44991 was elected leader, sending a full tablet report...
I20260928 21:20:49.722133 24874 ts_tablet_manager.cc:1428] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260928 21:20:49.723083 24876 ts_tablet_manager.cc:1428] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:20:49.723187 24589 heartbeater.cc:503] Master 127.5.18.254:44991 was elected leader, sending a full tablet report...
I20260928 21:20:49.725085 24875 raft_consensus.cc:348] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.725396 24875 raft_consensus.cc:374] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:49.725476 24875 raft_consensus.cc:729] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a0c553f923408a874dc948e49f7c09, State: Initialized, Role: FOLLOWER
I20260928 21:20:49.725997 24875 consensus_queue.cc:260] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:49.726655 24723 heartbeater.cc:503] Master 127.5.18.254:44991 was elected leader, sending a full tablet report...
I20260928 21:20:49.727066 24875 ts_tablet_manager.cc:1428] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 21:20:49.892544 24858 tablet.cc:2369] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:20:49.932054 24724 tablet.cc:2369] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:20:49.960580 24590 tablet.cc:2369] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:50.002708 24881 raft_consensus.cc:482] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:50.002912 24881 raft_consensus.cc:504] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:50.004079 24881 leader_election.cc:288] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805), c40b6ec1cea24808abd43d63001958f5 (127.5.18.195:43509)
I20260928 21:20:50.004812 24880 raft_consensus.cc:482] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:50.005121 24880 raft_consensus.cc:504] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:50.006896 24880 leader_election.cc:288] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805), 0202adbcb9954b3e945bec70aba62459 (127.5.18.193:34083)
I20260928 21:20:50.009008 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.004417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:38933 (local address 127.5.18.194:38805)
0928 21:20:50.004581 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.004586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.004788 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:20:50.005007 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.005011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.005060 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.005100 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.005584 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.006247 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.007913 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.008512 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.008614 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.008677 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.008823 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.008888 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:20:50.009028 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.004389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38805 (local address 127.5.18.193:38933)
0928 21:20:50.004682 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.004699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.004875 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.005256 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.005259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.005278 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.005452 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.005460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.006373 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.006376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.007778 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.007787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.008614 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.008618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.008712 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.008735 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.008802 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.008875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260928 21:20:50.009958 24677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "0202adbcb9954b3e945bec70aba62459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a0c553f923408a874dc948e49f7c09" is_pre_election: true
I20260928 21:20:50.010300 24677 raft_consensus.cc:2454] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0202adbcb9954b3e945bec70aba62459 in term 0.
I20260928 21:20:50.010782 24873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.004883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:38677 (local address 127.5.18.195:43509)
0928 21:20:50.004991 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.004995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.005225 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:20:50.005372 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.005376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.005425 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.005469 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.005881 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.006554 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.008232 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.009881 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.009989 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.010037 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.010555 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.010630 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1045,"server-negotiator.queue_time_us":38}
I20260928 21:20:50.010828 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.006497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34083 (local address 127.5.18.195:56661)
0928 21:20:50.006725 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.006738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.006815 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.007228 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.007230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.007245 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.007328 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.007332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.008205 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.008207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.009444 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.009453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.010335 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.010339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.010432 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.010453 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.010523 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.010598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":34}
I20260928 21:20:50.010847 24477 leader_election.cc:302] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0202adbcb9954b3e945bec70aba62459, 70a0c553f923408a874dc948e49f7c09; no voters:
I20260928 21:20:50.011125 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.004816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:43509 (local address 127.5.18.193:38677)
0928 21:20:50.005126 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.005139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.005249 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.005627 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.005630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.005645 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.005765 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.005771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.006667 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.006670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.008100 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.008110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.010097 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.010100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.010434 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.010454 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.010905 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.010984 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":229,"thread_start_us":103,"threads_started":1}
I20260928 21:20:50.011224 24881 raft_consensus.cc:2781] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:50.011301 24881 raft_consensus.cc:482] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:50.011343 24881 raft_consensus.cc:3037] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:50.011467 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.006912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:56661 (local address 127.5.18.193:34083)
0928 21:20:50.007010 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.007014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.007026 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:20:50.007063 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.007066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.007112 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.007192 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.007424 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.008098 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.009602 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.010223 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.010319 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.010360 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.011269 (+ 909us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.011336 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":32}
I20260928 21:20:50.011698 24886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.006497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:38805 (local address 127.5.18.195:35677)
0928 21:20:50.007547 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.007560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.007676 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.007978 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.007981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.007994 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.008098 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.008103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.009147 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.009150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.010351 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.010360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.011319 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.011322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.011414 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.011435 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.011509 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.011571 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":465,"thread_start_us":99,"threads_started":1}
I20260928 21:20:50.011847 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.006636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:35677 (local address 127.5.18.194:38805)
0928 21:20:50.007135 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.007138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.007620 (+ 482us) server_negotiation.cc:407] Connection header received
0928 21:20:50.007776 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.007779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.007820 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.007869 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.008217 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.009010 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.010472 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.011066 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.011153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.011212 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.011652 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.011729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":111,"threads_started":1}
I20260928 21:20:50.012180 24677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "c40b6ec1cea24808abd43d63001958f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a0c553f923408a874dc948e49f7c09" is_pre_election: true
I20260928 21:20:50.012315 24677 raft_consensus.cc:2454] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40b6ec1cea24808abd43d63001958f5 in term 0.
I20260928 21:20:50.012718 24745 leader_election.cc:302] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a0c553f923408a874dc948e49f7c09, c40b6ec1cea24808abd43d63001958f5; no voters:
I20260928 21:20:50.012768 24881 raft_consensus.cc:504] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:50.013026 24880 raft_consensus.cc:2781] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:50.013096 24881 leader_election.cc:288] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [CANDIDATE]: Term 1 election: Requested vote from peers 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805), c40b6ec1cea24808abd43d63001958f5 (127.5.18.195:43509)
I20260928 21:20:50.013106 24880 raft_consensus.cc:482] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:50.013154 24880 raft_consensus.cc:3037] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:50.013356 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "c40b6ec1cea24808abd43d63001958f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0202adbcb9954b3e945bec70aba62459" is_pre_election: true
I20260928 21:20:50.013541 24543 raft_consensus.cc:2379] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c40b6ec1cea24808abd43d63001958f5 in current term 1: Already voted for candidate 0202adbcb9954b3e945bec70aba62459 in this term.
I20260928 21:20:50.014261 24880 raft_consensus.cc:504] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:50.014420 24677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "0202adbcb9954b3e945bec70aba62459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a0c553f923408a874dc948e49f7c09"
I20260928 21:20:50.014545 24677 raft_consensus.cc:3037] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:50.014621 24880 leader_election.cc:288] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [CANDIDATE]: Term 1 election: Requested vote from peers 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805), 0202adbcb9954b3e945bec70aba62459 (127.5.18.193:34083)
I20260928 21:20:50.014768 24811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "0202adbcb9954b3e945bec70aba62459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40b6ec1cea24808abd43d63001958f5" is_pre_election: true
I20260928 21:20:50.014977 24811 raft_consensus.cc:2379] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0202adbcb9954b3e945bec70aba62459 in current term 1: Already voted for candidate c40b6ec1cea24808abd43d63001958f5 in this term.
I20260928 21:20:50.015048 24810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "0202adbcb9954b3e945bec70aba62459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40b6ec1cea24808abd43d63001958f5"
I20260928 21:20:50.015177 24810 raft_consensus.cc:2379] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0202adbcb9954b3e945bec70aba62459 in current term 1: Already voted for candidate c40b6ec1cea24808abd43d63001958f5 in this term.
I20260928 21:20:50.015146 24676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "c40b6ec1cea24808abd43d63001958f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a0c553f923408a874dc948e49f7c09"
I20260928 21:20:50.015852 24677 raft_consensus.cc:2454] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0202adbcb9954b3e945bec70aba62459 in term 1.
I20260928 21:20:50.015926 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8d4285d62cd4cb5bd7619eb3bb77390" candidate_uuid: "c40b6ec1cea24808abd43d63001958f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0202adbcb9954b3e945bec70aba62459"
I20260928 21:20:50.016052 24543 raft_consensus.cc:2379] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c40b6ec1cea24808abd43d63001958f5 in current term 1: Already voted for candidate 0202adbcb9954b3e945bec70aba62459 in this term.
I20260928 21:20:50.016194 24477 leader_election.cc:302] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0202adbcb9954b3e945bec70aba62459, 70a0c553f923408a874dc948e49f7c09; no voters: c40b6ec1cea24808abd43d63001958f5
I20260928 21:20:50.016304 24747 leader_election.cc:302] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c40b6ec1cea24808abd43d63001958f5; no voters: 0202adbcb9954b3e945bec70aba62459, 70a0c553f923408a874dc948e49f7c09
I20260928 21:20:50.016366 24881 raft_consensus.cc:2781] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:50.016440 24880 raft_consensus.cc:2726] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 21:20:50.016597 24881 raft_consensus.cc:686] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 LEADER]: Becoming Leader. State: Replica: 0202adbcb9954b3e945bec70aba62459, State: Running, Role: LEADER
I20260928 21:20:50.016938 24881 consensus_queue.cc:237] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } }
I20260928 21:20:50.020079 24396 catalog_manager.cc:5495] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0202adbcb9954b3e945bec70aba62459 (127.5.18.193). New cstate: current_term: 1 leader_uuid: "0202adbcb9954b3e945bec70aba62459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0202adbcb9954b3e945bec70aba62459" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 34083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 } health_report { overall_health: UNKNOWN } } }
I20260928 21:20:50.061244 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.057884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.5.18.193:34083)
0928 21:20:50.058017 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.058021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.058036 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:20:50.058129 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.058131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.058185 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.058269 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:50.058501 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.059002 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.059850 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.060065 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.060150 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.060441 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:50.060881 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:50.060986 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.061049 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.061103 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:20:50.067241 24677 raft_consensus.cc:1261] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 1 FOLLOWER]: Refusing update from remote peer 0202adbcb9954b3e945bec70aba62459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:50.067245 24810 raft_consensus.cc:1261] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Refusing update from remote peer 0202adbcb9954b3e945bec70aba62459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:50.067787 24881 consensus_queue.cc:1035] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a0c553f923408a874dc948e49f7c09" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 38805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:50.067981 24887 consensus_queue.cc:1035] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c40b6ec1cea24808abd43d63001958f5" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:50.075893 24892 mvcc.cc:205] Tried to move back new op lower bound from 7334422323468546048 to 7334422323271667712. Current Snapshot: MvccSnapshot[applied={T|T < 7334422323468546048}]
I20260928 21:20:50.083838 24396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:50.084033 24396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:50.086431 24396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 205eda36e9704747b2cc5d88b6ee015d: Sending DeleteTablet for 3 replicas of tablet d8d4285d62cd4cb5bd7619eb3bb77390
I20260928 21:20:50.087178 24523 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:50 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:34648
I20260928 21:20:50.087450 24657 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:50 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:33764
I20260928 21:20:50.087461 24791 tablet_service.cc:1514] Processing DeleteTablet for tablet d8d4285d62cd4cb5bd7619eb3bb77390 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:50 UTC) from {username='kudu', principal='kudu/127.5.18.254@KRBTEST.COM'} at 127.0.0.1:45718
I20260928 21:20:50.087857 24899 tablet_replica.cc:330] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459: stopping tablet replica
I20260928 21:20:50.088184 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24460
I20260928 21:20:50.088186 24899 raft_consensus.cc:2229] T d8d4285d62cd4cb5bd7619eb3bb77390 P 0202adbcb9954b3e945bec70aba62459 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:50.089490 24900 tablet_replica.cc:330] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: stopping tablet replica
I20260928 21:20:50.089838 24900 raft_consensus.cc:2229] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:50.090163 24900 raft_consensus.cc:2258] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:50.090551 24901 tablet_replica.cc:330] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: stopping tablet replica
I20260928 21:20:50.090832 24901 raft_consensus.cc:2229] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:50.091128 24901 raft_consensus.cc:2258] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:50.091584 24900 ts_tablet_manager.cc:1905] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:50.092519 24901 ts_tablet_manager.cc:1905] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:50.093863 24900 ts_tablet_manager.cc:1918] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:50.093950 24900 log.cc:1196] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/d8d4285d62cd4cb5bd7619eb3bb77390
I20260928 21:20:50.094211 24900 ts_tablet_manager.cc:1939] T d8d4285d62cd4cb5bd7619eb3bb77390 P c40b6ec1cea24808abd43d63001958f5: Deleting consensus metadata
I20260928 21:20:50.095012 24901 ts_tablet_manager.cc:1918] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:50.095101 24901 log.cc:1196] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790630206151656-5195-0/minicluster-data/ts-1/wal/wals/d8d4285d62cd4cb5bd7619eb3bb77390
I20260928 21:20:50.095373 24901 ts_tablet_manager.cc:1939] T d8d4285d62cd4cb5bd7619eb3bb77390 P 70a0c553f923408a874dc948e49f7c09: Deleting consensus metadata
I20260928 21:20:50.095373 24383 catalog_manager.cc:4841] TS c40b6ec1cea24808abd43d63001958f5 (127.5.18.195:43509): tablet d8d4285d62cd4cb5bd7619eb3bb77390 (table test-table [id=1a2930381b2a4fcb933970b57db9e0ce]) successfully deleted
I20260928 21:20:50.096463 24383 catalog_manager.cc:4841] TS 70a0c553f923408a874dc948e49f7c09 (127.5.18.194:38805): tablet d8d4285d62cd4cb5bd7619eb3bb77390 (table test-table [id=1a2930381b2a4fcb933970b57db9e0ce]) successfully deleted
W20260928 21:20:50.096910 24385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:34083 (error 108)
I20260928 21:20:50.097091 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24594
I20260928 21:20:50.098631 24870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.098120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:34083 (local address 127.0.0.1:34662)
0928 21:20:50.098471 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.098555 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
W20260928 21:20:50.098790 24385 catalog_manager.cc:4584] TS 0202adbcb9954b3e945bec70aba62459 (127.5.18.193:34083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8d4285d62cd4cb5bd7619eb3bb77390: Network error: Client connection negotiation failed: client connection to 127.5.18.193:34083: connect: Connection refused (error 111)
I20260928 21:20:50.103192 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24728
I20260928 21:20:50.108227 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24366
2026-09-28T21:20:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2473 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T21:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:50Z Disabled control of system clock
I20260928 21:20:50.136407 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34765
--webserver_interface=127.5.18.254
--webserver_port=0
--builtin_ntp_servers=127.5.18.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:34765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:50.245203 24906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:50.245478 24906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:50.245513 24906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:50.249485 24906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:50.249579 24906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:50.249595 24906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:50.249610 24906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:50.249624 24906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:50.254500 24906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:34765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:34765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:50.255575 24906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:50.256939 24906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:50.265483 24912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:50.265519 24906 server_base.cc:1034] running on GCE node
W20260928 21:20:50.265465 24914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.265455 24911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:50.266245 24906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:50.267387 24906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:50.268555 24906 hybrid_clock.cc:648] HybridClock initialized: now 1790630450268535 us; error 38 us; skew 500 ppm
I20260928 21:20:50.270500 24906 webserver.cc:460] Webserver started at http://127.5.18.254:39521/ using document root <none> and password file <none>
I20260928 21:20:50.271063 24906 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:50.271128 24906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:50.271335 24906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:50.273352 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "ec29151e3e944ec3b65746c34f6720fa"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "700f44f99ac46da290a8a53af45a1c0b"
server_key_iv: "4dd49993257643d80234ed536af65ae0"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.274039 24906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "ec29151e3e944ec3b65746c34f6720fa"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "700f44f99ac46da290a8a53af45a1c0b"
server_key_iv: "4dd49993257643d80234ed536af65ae0"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.277740 24906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:20:50.280133 24920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.281371 24906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:50.281491 24906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "ec29151e3e944ec3b65746c34f6720fa"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "700f44f99ac46da290a8a53af45a1c0b"
server_key_iv: "4dd49993257643d80234ed536af65ae0"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.281632 24906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:50.294087 24906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:50.294839 24906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:50.295022 24906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:50.301510 24906 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:34765
I20260928 21:20:50.301527 24972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:34765 every 8 connection(s)
I20260928 21:20:50.302475 24906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:50.305670 24973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:50.311652 24973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: Bootstrap starting.
I20260928 21:20:50.312090 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24906
I20260928 21:20:50.312228 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:50.312489 5195 external_mini_cluster.cc:1428] Setting key 5a256ed3b0ee4788ba828f10de703621
I20260928 21:20:50.314388 24973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:50.315119 24973 log.cc:824] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:50.317210 24973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: No bootstrap required, opened a new log
I20260928 21:20:50.321272 24973 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } }
I20260928 21:20:50.321501 24973 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:50.321532 24973 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec29151e3e944ec3b65746c34f6720fa, State: Initialized, Role: FOLLOWER
I20260928 21:20:50.321789 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.313933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.5.18.254:34765)
0928 21:20:50.314585 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.314601 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.314637 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:20:50.315324 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.315343 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.315740 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.316015 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:50.316361 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.317333 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.318507 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.319010 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.319124 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.319998 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:50.320003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:50.320015 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:50.320405 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:50.320462 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:50.320481 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:50.320555 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.320693 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.320951 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260928 21:20:50.322012 24973 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } }
I20260928 21:20:50.322144 24973 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:50.322217 24973 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:50.322311 24973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:50.323190 24973 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } }
I20260928 21:20:50.323510 24973 leader_election.cc:302] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec29151e3e944ec3b65746c34f6720fa; no voters:
I20260928 21:20:50.323820 24973 leader_election.cc:288] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:50.323935 24978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:50.324136 24978 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [term 1 LEADER]: Becoming Leader. State: Replica: ec29151e3e944ec3b65746c34f6720fa, State: Running, Role: LEADER
I20260928 21:20:50.324438 24978 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } }
I20260928 21:20:50.324991 24973 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:50.326129 24979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec29151e3e944ec3b65746c34f6720fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } } }
I20260928 21:20:50.326148 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec29151e3e944ec3b65746c34f6720fa. Latest consensus state: current_term: 1 leader_uuid: "ec29151e3e944ec3b65746c34f6720fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec29151e3e944ec3b65746c34f6720fa" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 34765 } } }
I20260928 21:20:50.326277 24979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:50.326282 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:50.326611 24987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:50.329720 24987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:50.335539 24987 catalog_manager.cc:1347] Generated new cluster ID: 80208e46739c4f45a1fd18df38c06cef
I20260928 21:20:50.335629 24987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:50.342902 24987 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:50.344201 24987 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:50.355448 24987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: Generated new TSK 0
I20260928 21:20:50.356369 24987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:50.363649 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.193:0
--local_ip_for_outbound_sockets=127.5.18.193
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--builtin_ntp_servers=127.5.18.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:20:50.475701 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:50.475984 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:50.476024 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:50.480162 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:50.480225 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:50.480283 24997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.193
I20260928 21:20:50.485212 24997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.18.193
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.193
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:50.486351 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:50.487712 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:50.497431 25002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.497416 25005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.497509 25003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:50.497779 24997 server_base.cc:1034] running on GCE node
I20260928 21:20:50.498198 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:50.498855 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:50.500005 24997 hybrid_clock.cc:648] HybridClock initialized: now 1790630450499981 us; error 38 us; skew 500 ppm
I20260928 21:20:50.502043 24997 webserver.cc:460] Webserver started at http://127.5.18.193:34551/ using document root <none> and password file <none>
I20260928 21:20:50.502616 24997 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:50.502682 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:50.502895 24997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:50.505079 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data/instance:
uuid: "0eaad83153b24391879634ef53f86929"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "a1af58d9a1d1289f8841e9f522bfd295"
server_key_iv: "935551432e493a2437d7b358c655271b"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.505729 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance:
uuid: "0eaad83153b24391879634ef53f86929"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "a1af58d9a1d1289f8841e9f522bfd295"
server_key_iv: "935551432e493a2437d7b358c655271b"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.509497 24997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:20:50.512054 25011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.513330 24997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:20:50.513444 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal
uuid: "0eaad83153b24391879634ef53f86929"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "a1af58d9a1d1289f8841e9f522bfd295"
server_key_iv: "935551432e493a2437d7b358c655271b"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.513559 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:50.540174 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:50.540906 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:50.541082 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:50.541688 24997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:50.542627 24997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:50.542675 24997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.542726 24997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:50.542762 24997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.551367 24997 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.193:37187
I20260928 21:20:50.551419 25124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.193:37187 every 8 connection(s)
I20260928 21:20:50.552295 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/data/info.pb
I20260928 21:20:50.559990 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 24997
I20260928 21:20:50.560127 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-0/wal/instance
I20260928 21:20:50.560406 5195 external_mini_cluster.cc:1428] Setting key 8b8572f38bfb02b5a26bc3df0895f8bf
I20260928 21:20:50.561785 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.554123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.193:60143 (local address 127.5.18.254:34765)
0928 21:20:50.554248 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.554252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.555346 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 21:20:50.556789 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.556792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.556862 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.556936 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:50.557906 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.558436 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.559567 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.559879 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.559995 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.561239 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:50.561242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:50.561245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:50.561257 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:50.561280 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:50.561290 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:50.561364 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.561555 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.561647 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:20:50.562486 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.194:0
--local_ip_for_outbound_sockets=127.5.18.194
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--builtin_ntp_servers=127.5.18.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:50.562798 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.554541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34765 (local address 127.5.18.193:60143)
0928 21:20:50.555193 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.555234 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.556558 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.557142 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.557150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.557242 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:50.557768 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.557781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.558557 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.558559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.559456 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.559465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.560072 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.560077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.560296 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.560978 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:50.560980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:50.561037 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:50.561109 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:50.561409 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:50.561419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:50.561424 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.561544 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.561903 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260928 21:20:50.564167 25125 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34765
I20260928 21:20:50.564448 25125 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:50.565011 25125 heartbeater.cc:511] Master 127.5.18.254:34765 requested a full tablet report, sending...
I20260928 21:20:50.566581 24937 ts_manager.cc:194] Registered new tserver with Master: 0eaad83153b24391879634ef53f86929 (127.5.18.193:37187)
I20260928 21:20:50.567718 24937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.193:60143
W20260928 21:20:50.676306 25128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:50.676622 25128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:50.676666 25128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:50.680955 25128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:50.681034 25128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:50.681094 25128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.194
I20260928 21:20:50.686198 25128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.18.194
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.194
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:50.687372 25128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:50.688895 25128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:50.698997 25136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.698993 25134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.699003 25133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:50.699214 25128 server_base.cc:1034] running on GCE node
I20260928 21:20:50.699702 25128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:50.700408 25128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:50.701594 25128 hybrid_clock.cc:648] HybridClock initialized: now 1790630450701548 us; error 37 us; skew 500 ppm
I20260928 21:20:50.703616 25128 webserver.cc:460] Webserver started at http://127.5.18.194:46275/ using document root <none> and password file <none>
I20260928 21:20:50.704200 25128 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:50.704280 25128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:50.704485 25128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:50.706667 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data/instance:
uuid: "76891b0cd0e4471b86b55af36afa682a"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "73e96fd148e65eb7c988b10d554e656a"
server_key_iv: "6ba224b6d4fd403caec0b696384a369a"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.707227 25128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance:
uuid: "76891b0cd0e4471b86b55af36afa682a"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "73e96fd148e65eb7c988b10d554e656a"
server_key_iv: "6ba224b6d4fd403caec0b696384a369a"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.711169 25128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:20:50.713752 25142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.715040 25128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:50.715161 25128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal
uuid: "76891b0cd0e4471b86b55af36afa682a"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "73e96fd148e65eb7c988b10d554e656a"
server_key_iv: "6ba224b6d4fd403caec0b696384a369a"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.715279 25128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:50.734844 25128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:50.735589 25128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:50.735781 25128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:50.736404 25128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:50.737392 25128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:50.737440 25128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.737494 25128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:50.737520 25128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.745872 25128 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.194:35759
I20260928 21:20:50.745899 25255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.194:35759 every 8 connection(s)
I20260928 21:20:50.746780 25128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/data/info.pb
I20260928 21:20:50.749867 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25128
I20260928 21:20:50.750001 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-1/wal/instance
I20260928 21:20:50.750348 5195 external_mini_cluster.cc:1428] Setting key 59c345fb62cc749de3a29b277f644f40
I20260928 21:20:50.752467 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.18.195:0
--local_ip_for_outbound_sockets=127.5.18.195
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--builtin_ntp_servers=127.5.18.212:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:20:50.756645 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.748573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.194:33307 (local address 127.5.18.254:34765)
0928 21:20:50.748745 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.748750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.749940 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 21:20:50.751516 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.751519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.751566 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.751652 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:50.752850 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.753376 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.754687 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.754912 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.755001 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.756162 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:50.756165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:50.756167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:50.756182 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:50.756202 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:50.756210 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:50.756276 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.756416 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.756488 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:20:50.757558 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.749009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34765 (local address 127.5.18.194:33307)
0928 21:20:50.749732 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.749799 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.751306 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.751851 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.751861 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.751972 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:50.752688 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.752708 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.753501 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.753505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.754557 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.754567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.755065 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.755069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.755232 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.755887 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:50.755889 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:50.755936 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:50.756009 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:50.756287 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:50.756297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:50.756303 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.756409 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.756715 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260928 21:20:50.758816 25256 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34765
I20260928 21:20:50.759064 25256 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:50.759635 25256 heartbeater.cc:511] Master 127.5.18.254:34765 requested a full tablet report, sending...
I20260928 21:20:50.760754 24937 ts_manager.cc:194] Registered new tserver with Master: 76891b0cd0e4471b86b55af36afa682a (127.5.18.194:35759)
I20260928 21:20:50.761602 24937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.194:33307
W20260928 21:20:50.863451 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:50.863736 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:50.863776 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:50.867949 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:50.868017 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:50.868074 25259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.18.195
I20260928 21:20:50.873046 25259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.18.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.18.195
--webserver_port=0
--tserver_master_addrs=127.5.18.254:34765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.18.195
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:50.874191 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:50.875521 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:20:50.884950 25259 server_base.cc:1034] running on GCE node
W20260928 21:20:50.884899 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.884900 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:50.884899 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:50.885659 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:50.886312 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:50.887478 25259 hybrid_clock.cc:648] HybridClock initialized: now 1790630450887459 us; error 34 us; skew 500 ppm
I20260928 21:20:50.889319 25259 webserver.cc:460] Webserver started at http://127.5.18.195:37651/ using document root <none> and password file <none>
I20260928 21:20:50.889897 25259 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:50.889959 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:50.890123 25259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:50.892143 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data/instance:
uuid: "69fbd0b143a84ba28340dedcd8c8abda"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "939d7672b8b4240d0a5f9aa5606c9585"
server_key_iv: "9d10ff30410ecba1813aa50aa59985bd"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.892632 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance:
uuid: "69fbd0b143a84ba28340dedcd8c8abda"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "939d7672b8b4240d0a5f9aa5606c9585"
server_key_iv: "9d10ff30410ecba1813aa50aa59985bd"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.896401 25259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:20:50.898950 25273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.900187 25259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:20:50.900290 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal
uuid: "69fbd0b143a84ba28340dedcd8c8abda"
format_stamp: "Formatted at 2026-09-28 21:20:50 on dist-test-slave-rc9f"
server_key: "939d7672b8b4240d0a5f9aa5606c9585"
server_key_iv: "9d10ff30410ecba1813aa50aa59985bd"
server_key_version: "encryptionkey@0"
I20260928 21:20:50.900406 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:50.929966 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:50.930683 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:50.930873 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:50.931470 25259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:20:50.932478 25259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:20:50.932525 25259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.932565 25259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:20:50.932580 25259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:50.941134 25259 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.195:41455
I20260928 21:20:50.941164 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.195:41455 every 8 connection(s)
I20260928 21:20:50.942346 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/data/info.pb
I20260928 21:20:50.949538 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25259
I20260928 21:20:50.949684 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal/instance
I20260928 21:20:50.949985 5195 external_mini_cluster.cc:1428] Setting key b9b75c58929e0e272075b08f4a46bfaf
I20260928 21:20:50.958359 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.951130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:57127 (local address 127.5.18.254:34765)
0928 21:20:50.951290 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.951294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.951311 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:20:50.952400 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.952404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.952464 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.952514 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:50.953896 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.954725 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.956173 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.956399 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.956492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.957754 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:50.957758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:50.957760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:50.957780 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:50.957803 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:50.957812 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:50.957888 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.958144 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.958224 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:20:50.959352 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.949507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:34765 (local address 127.5.18.195:57127)
0928 21:20:50.950428 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.950490 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.952106 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.952969 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.952980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.953102 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:20:50.953749 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.953769 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.954868 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.954872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.956047 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.956059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.956538 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.956540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.956697 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.957316 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:20:50.957318 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:20:50.957372 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:20:50.957456 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:20:50.957948 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:20:50.957956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:20:50.957961 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.958081 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.958361 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":159,"threads_started":1}
I20260928 21:20:50.960704 25387 heartbeater.cc:347] Connected to a master server at 127.5.18.254:34765
I20260928 21:20:50.960976 25387 heartbeater.cc:464] Registering TS with master...
I20260928 21:20:50.961550 25387 heartbeater.cc:511] Master 127.5.18.254:34765 requested a full tablet report, sending...
I20260928 21:20:50.962987 24937 ts_manager.cc:194] Registered new tserver with Master: 69fbd0b143a84ba28340dedcd8c8abda (127.5.18.195:41455)
I20260928 21:20:50.963716 24937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.18.195:57127
I20260928 21:20:50.964444 5195 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:20:50.970588 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.966837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.5.18.254:34765)
0928 21:20:50.966984 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.966989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.967116 (+ 127us) server_negotiation.cc:407] Connection header received
0928 21:20:50.967242 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.967245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.967303 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.967369 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:50.967637 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.968162 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.969002 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.969233 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.969331 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.970174 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:50.970177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:50.970180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:50.970196 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:50.970217 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:50.970226 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:50.970292 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.970372 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.970434 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:20:50.973790 24936 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:20:50.976783 24936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:20:50.991035 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.985261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:37187 (local address 127.0.0.1:44382)
0928 21:20:50.986085 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.986136 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.986274 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.987071 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.987077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.987098 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.987244 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.987252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988307 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.988310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.989392 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.989404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.990478 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.990482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.990591 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.990627 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.990750 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.990878 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":95,"threads_started":1}
I20260928 21:20:50.991040 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.985388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.5.18.193:37187)
0928 21:20:50.985772 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.985785 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.986413 (+ 628us) server_negotiation.cc:407] Connection header received
0928 21:20:50.986499 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.986504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.986753 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.986895 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.987360 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988202 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.989532 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.990380 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.990468 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.990675 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.990782 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.990882 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":132,"threads_started":1}
I20260928 21:20:50.991516 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.985649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:35759 (local address 127.0.0.1:47952)
0928 21:20:50.986254 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.986275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.986361 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.987276 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.987278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.987307 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.987488 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.987496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988919 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.988923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.990130 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.990139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.991152 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.991156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.991241 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.991263 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.991312 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.991377 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":107,"spinlock_wait_cycles":7552,"thread_start_us":113,"threads_started":1}
I20260928 21:20:50.991710 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.986037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.195:41455 (local address 127.0.0.1:44918)
0928 21:20:50.986517 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:50.986528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:50.986602 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:50.987273 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:50.987276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:50.987305 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:50.987521 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.987531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988919 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.988923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.990217 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:50.990228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.991362 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.991365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:50.991453 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.991472 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:20:50.991533 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 21:20:50.991592 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":160,"spinlock_wait_cycles":7168,"thread_start_us":90,"threads_started":1}
I20260928 21:20:50.992203 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.986531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.5.18.195:41455)
0928 21:20:50.986827 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.986843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.986865 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:20:50.986935 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.986939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.987117 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.987519 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.987652 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988800 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.990322 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.991034 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.991122 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.991308 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.991944 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.992049 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":118,"threads_started":1}
I20260928 21:20:50.992228 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:50.986117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.5.18.194:35759)
0928 21:20:50.986839 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:50.986851 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:50.986873 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:20:50.986942 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:50.986947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:50.987143 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:50.987302 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:50.987614 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.988792 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.990239 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:50.991017 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:50.991781 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:50.991968 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:50.992009 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:20:50.992086 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":117,"threads_started":1}
I20260928 21:20:50.994576 25059 tablet_service.cc:1467] Processing CreateTablet for tablet 4d62ed88869c438194594136e7c66ec5 (DEFAULT_TABLE table=test-table [id=5137e623dd0548548a7ce3c3821d794e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:50.994988 25321 tablet_service.cc:1467] Processing CreateTablet for tablet 4d62ed88869c438194594136e7c66ec5 (DEFAULT_TABLE table=test-table [id=5137e623dd0548548a7ce3c3821d794e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:50.995800 25059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d62ed88869c438194594136e7c66ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:50.995975 25321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d62ed88869c438194594136e7c66ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:50.995707 25190 tablet_service.cc:1467] Processing CreateTablet for tablet 4d62ed88869c438194594136e7c66ec5 (DEFAULT_TABLE table=test-table [id=5137e623dd0548548a7ce3c3821d794e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:20:50.996603 25190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d62ed88869c438194594136e7c66ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:51.002884 25404 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: Bootstrap starting.
I20260928 21:20:51.004274 25406 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Bootstrap starting.
I20260928 21:20:51.005741 25404 tablet_bootstrap.cc:654] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:51.005967 25405 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Bootstrap starting.
I20260928 21:20:51.006184 25406 tablet_bootstrap.cc:654] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:51.006534 25404 log.cc:824] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:51.006973 25406 log.cc:824] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:51.008107 25405 tablet_bootstrap.cc:654] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:51.008795 25404 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: No bootstrap required, opened a new log
I20260928 21:20:51.008917 25405 log.cc:824] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:51.008981 25404 ts_tablet_manager.cc:1397] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:20:51.009475 25406 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: No bootstrap required, opened a new log
I20260928 21:20:51.009719 25406 ts_tablet_manager.cc:1397] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:20:51.011317 25405 tablet_bootstrap.cc:492] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: No bootstrap required, opened a new log
I20260928 21:20:51.011559 25405 ts_tablet_manager.cc:1397] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:20:51.012430 25404 raft_consensus.cc:348] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.012670 25404 raft_consensus.cc:374] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:51.012723 25404 raft_consensus.cc:729] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76891b0cd0e4471b86b55af36afa682a, State: Initialized, Role: FOLLOWER
I20260928 21:20:51.013167 25404 consensus_queue.cc:260] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.013888 25256 heartbeater.cc:503] Master 127.5.18.254:34765 was elected leader, sending a full tablet report...
I20260928 21:20:51.014181 25404 ts_tablet_manager.cc:1428] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:20:51.014004 25406 raft_consensus.cc:348] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.014249 25406 raft_consensus.cc:374] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:51.014302 25406 raft_consensus.cc:729] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fbd0b143a84ba28340dedcd8c8abda, State: Initialized, Role: FOLLOWER
I20260928 21:20:51.014768 25406 consensus_queue.cc:260] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.015782 25406 ts_tablet_manager.cc:1428] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:20:51.016145 25387 heartbeater.cc:503] Master 127.5.18.254:34765 was elected leader, sending a full tablet report...
I20260928 21:20:51.016278 25405 raft_consensus.cc:348] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.016566 25405 raft_consensus.cc:374] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:51.016621 25405 raft_consensus.cc:729] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eaad83153b24391879634ef53f86929, State: Initialized, Role: FOLLOWER
I20260928 21:20:51.017050 25405 consensus_queue.cc:260] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.017692 25125 heartbeater.cc:503] Master 127.5.18.254:34765 was elected leader, sending a full tablet report...
I20260928 21:20:51.018014 25405 ts_tablet_manager.cc:1428] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260928 21:20:51.039315 25411 raft_consensus.cc:482] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:20:51.039528 25411 raft_consensus.cc:504] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.040721 25411 leader_election.cc:288] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eaad83153b24391879634ef53f86929 (127.5.18.193:37187), 76891b0cd0e4471b86b55af36afa682a (127.5.18.194:35759)
I20260928 21:20:51.044486 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.041102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:35133 (local address 127.5.18.194:35759)
0928 21:20:51.041237 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:51.041242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:51.041498 (+ 256us) server_negotiation.cc:407] Connection header received
0928 21:20:51.041622 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:51.041625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:51.041678 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:51.041764 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:51.042019 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.042539 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.043534 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.043935 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.044042 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:51.044088 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:51.044310 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 21:20:51.044381 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:20:51.044486 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.041141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.18.195:46313 (local address 127.5.18.193:37187)
0928 21:20:51.041271 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:51.041276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:51.041290 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:20:51.041343 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:51.041346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:51.041397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:51.041482 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:20:51.041827 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.042416 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.043513 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.043913 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.044022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:51.044078 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:51.044292 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:20:51.044366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:20:51.044486 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.040941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:37187 (local address 127.5.18.195:46313)
0928 21:20:51.041055 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:51.041071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:51.041166 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:51.041519 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:51.041523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:51.041548 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:51.041712 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:51.041720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.042524 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.042528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:51.043382 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:51.043393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.044028 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.044032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:51.044161 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:51.044192 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:20:51.044248 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:20:51.044327 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":6}
I20260928 21:20:51.044507 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.040999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:35759 (local address 127.5.18.195:35133)
0928 21:20:51.041402 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:51.041415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:20:51.041505 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:20:51.041793 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:20:51.041796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:20:51.041814 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:20:51.041920 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:51.041926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.042642 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.042645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:51.043409 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:20:51.043417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.044047 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.044050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:20:51.044173 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:51.044210 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:20:51.044279 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:20:51.044347 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":38,"thread_start_us":123,"threads_started":1}
I20260928 21:20:51.045202 25210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d62ed88869c438194594136e7c66ec5" candidate_uuid: "69fbd0b143a84ba28340dedcd8c8abda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76891b0cd0e4471b86b55af36afa682a" is_pre_election: true
I20260928 21:20:51.045200 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d62ed88869c438194594136e7c66ec5" candidate_uuid: "69fbd0b143a84ba28340dedcd8c8abda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eaad83153b24391879634ef53f86929" is_pre_election: true
I20260928 21:20:51.045497 25079 raft_consensus.cc:2454] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fbd0b143a84ba28340dedcd8c8abda in term 0.
I20260928 21:20:51.045496 25210 raft_consensus.cc:2454] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fbd0b143a84ba28340dedcd8c8abda in term 0.
I20260928 21:20:51.046005 25276 leader_election.cc:302] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fbd0b143a84ba28340dedcd8c8abda, 76891b0cd0e4471b86b55af36afa682a; no voters:
I20260928 21:20:51.046278 25411 raft_consensus.cc:2781] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:20:51.046365 25411 raft_consensus.cc:482] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:20:51.046409 25411 raft_consensus.cc:3037] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:51.047586 25411 raft_consensus.cc:504] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
I20260928 21:20:51.047957 25411 leader_election.cc:288] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [CANDIDATE]: Term 1 election: Requested vote from peers 0eaad83153b24391879634ef53f86929 (127.5.18.193:37187), 76891b0cd0e4471b86b55af36afa682a (127.5.18.194:35759)
I20260928 21:20:51.048310 25210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d62ed88869c438194594136e7c66ec5" candidate_uuid: "69fbd0b143a84ba28340dedcd8c8abda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76891b0cd0e4471b86b55af36afa682a"
I20260928 21:20:51.048311 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d62ed88869c438194594136e7c66ec5" candidate_uuid: "69fbd0b143a84ba28340dedcd8c8abda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eaad83153b24391879634ef53f86929"
I20260928 21:20:51.048421 25210 raft_consensus.cc:3037] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:51.048434 25079 raft_consensus.cc:3037] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:51.049432 25210 raft_consensus.cc:2454] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fbd0b143a84ba28340dedcd8c8abda in term 1.
I20260928 21:20:51.049453 25079 raft_consensus.cc:2454] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fbd0b143a84ba28340dedcd8c8abda in term 1.
I20260928 21:20:51.049744 25276 leader_election.cc:302] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fbd0b143a84ba28340dedcd8c8abda, 76891b0cd0e4471b86b55af36afa682a; no voters:
I20260928 21:20:51.049922 25411 raft_consensus.cc:2781] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:51.050148 25411 raft_consensus.cc:686] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 1 LEADER]: Becoming Leader. State: Replica: 69fbd0b143a84ba28340dedcd8c8abda, State: Running, Role: LEADER
I20260928 21:20:51.050458 25411 consensus_queue.cc:237] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } }
W20260928 21:20:51.053453 25126 tablet.cc:2369] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:20:51.053722 24937 catalog_manager.cc:5495] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fbd0b143a84ba28340dedcd8c8abda (127.5.18.195). New cstate: current_term: 1 leader_uuid: "69fbd0b143a84ba28340dedcd8c8abda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69fbd0b143a84ba28340dedcd8c8abda" member_type: VOTER last_known_addr { host: "127.5.18.195" port: 41455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 } health_report { overall_health: UNKNOWN } } }
I20260928 21:20:51.072085 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.068573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.5.18.195:41455)
0928 21:20:51.068765 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:51.068769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:51.068782 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:20:51.068859 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:51.068861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:51.068912 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:51.069002 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:20:51.069264 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.069805 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.070673 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:51.070887 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:51.070972 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:51.071253 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:20:51.071704 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:20:51.071811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:51.071872 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:20:51.071926 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 21:20:51.078370 25079 raft_consensus.cc:1261] T 4d62ed88869c438194594136e7c66ec5 P 0eaad83153b24391879634ef53f86929 [term 1 FOLLOWER]: Refusing update from remote peer 69fbd0b143a84ba28340dedcd8c8abda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:51.078370 25210 raft_consensus.cc:1261] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 1 FOLLOWER]: Refusing update from remote peer 69fbd0b143a84ba28340dedcd8c8abda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:20:51.079221 25411 consensus_queue.cc:1035] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eaad83153b24391879634ef53f86929" member_type: VOTER last_known_addr { host: "127.5.18.193" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:51.079445 25414 consensus_queue.cc:1035] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [LEADER]: Connected to new peer: Peer: permanent_uuid: "76891b0cd0e4471b86b55af36afa682a" member_type: VOTER last_known_addr { host: "127.5.18.194" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:20:51.086374 25417 mvcc.cc:205] Tried to move back new op lower bound from 7334422327608406016 to 7334422327505268736. Current Snapshot: MvccSnapshot[applied={T|T < 7334422327608406016}]
I20260928 21:20:51.087095 25419 mvcc.cc:205] Tried to move back new op lower bound from 7334422327608406016 to 7334422327505268736. Current Snapshot: MvccSnapshot[applied={T|T < 7334422327608406016}]
I20260928 21:20:51.088634 25418 mvcc.cc:205] Tried to move back new op lower bound from 7334422327608406016 to 7334422327505268736. Current Snapshot: MvccSnapshot[applied={T|T < 7334422327608406016}]
I20260928 21:20:51.095947 24937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:51.096165 24937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:20:51.098949 24937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec29151e3e944ec3b65746c34f6720fa: Sending DeleteTablet for 3 replicas of tablet 4d62ed88869c438194594136e7c66ec5
I20260928 21:20:51.099951 25321 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d62ed88869c438194594136e7c66ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:51 UTC) from {username='slave'} at 127.0.0.1:44918
I20260928 21:20:51.099946 25190 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d62ed88869c438194594136e7c66ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:51 UTC) from {username='slave'} at 127.0.0.1:47952
I20260928 21:20:51.100170 25059 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d62ed88869c438194594136e7c66ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:20:51 UTC) from {username='slave'} at 127.0.0.1:44382
I20260928 21:20:51.100651 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24997
I20260928 21:20:51.102772 25426 tablet_replica.cc:330] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: stopping tablet replica
I20260928 21:20:51.103147 25426 raft_consensus.cc:2229] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:20:51.103560 25426 raft_consensus.cc:2258] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:51.104043 25427 tablet_replica.cc:330] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a: stopping tablet replica
I20260928 21:20:51.104415 25427 raft_consensus.cc:2229] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:20:51.104766 25427 raft_consensus.cc:2258] T 4d62ed88869c438194594136e7c66ec5 P 76891b0cd0e4471b86b55af36afa682a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:20:51.105079 25426 ts_tablet_manager.cc:1905] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:20:51.107295 25426 ts_tablet_manager.cc:1918] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:20:51.107386 25426 log.cc:1196] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Deleting WAL directory at /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790630206151656-5195-0/minicluster-data/ts-2/wal/wals/4d62ed88869c438194594136e7c66ec5
I20260928 21:20:51.107640 25426 ts_tablet_manager.cc:1939] T 4d62ed88869c438194594136e7c66ec5 P 69fbd0b143a84ba28340dedcd8c8abda: Deleting consensus metadata
I20260928 21:20:51.108768 24922 catalog_manager.cc:4841] TS 69fbd0b143a84ba28340dedcd8c8abda (127.5.18.195:41455): tablet 4d62ed88869c438194594136e7c66ec5 (table test-table [id=5137e623dd0548548a7ce3c3821d794e]) successfully deleted
W20260928 21:20:51.110049 24922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.18.193:37187 (error 108)
I20260928 21:20:51.110319 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25128
I20260928 21:20:51.111524 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.111267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.193:37187 (local address 127.0.0.1:44388)
0928 21:20:51.111382 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:51.111447 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.193:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 21:20:51.111742 24922 catalog_manager.cc:4584] TS 0eaad83153b24391879634ef53f86929 (127.5.18.193:37187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d62ed88869c438194594136e7c66ec5: Network error: Client connection negotiation failed: client connection to 127.5.18.193:37187: connect: Connection refused (error 111)
I20260928 21:20:51.117292 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25259
I20260928 21:20:51.118829 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:51.117510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.194:35759 (local address 127.0.0.1:47956)
0928 21:20:51.118678 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 21:20:51.118738 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.194:35759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1069}
I20260928 21:20:51.122476 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 24906
2026-09-28T21:20:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1015 ms)
[----------] 6 tests from EncryptionPolicyTest (10392 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:51 dist-test-slave-rc9f krb5kdc[25432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:51 dist-test-slave-rc9f krb5kdc[25432](info): set up 2 sockets
Sep 28 21:20:51 dist-test-slave-rc9f krb5kdc[25432](info): commencing operation
krb5kdc: starting...
W20260928 21:20:52.077765 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:52.197204 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:52.307677 25445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:52.307976 25445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:52.308019 25445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:52.312167 25445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:52.312251 25445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:52.312270 25445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:52.312287 25445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:52.312307 25445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:52.317939 25445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:52.319033 25445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:52.320005 25445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:52.328713 25453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:52.328910 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:52.328924 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:52.329118 25445 server_base.cc:1034] running on GCE node
I20260928 21:20:52.329710 25445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:52.330808 25445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:52.331990 25445 hybrid_clock.cc:648] HybridClock initialized: now 1790630452331969 us; error 36 us; skew 500 ppm
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:52.334913 25445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:52.335871 25445 webserver.cc:460] Webserver started at http://127.0.0.1:37979/ using document root <none> and password file <none>
I20260928 21:20:52.336472 25445 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:52.336544 25445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:52.336753 25445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:52.338866 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "21acf65af8504fe8b67bff199b1a1cb3"
format_stamp: "Formatted at 2026-09-28 21:20:52 on dist-test-slave-rc9f"
server_key: "fc7358813923e1e9681c656d44b8255f"
server_key_iv: "10c18bcde8e2dab5398040549b6de4df"
server_key_version: "encryptionkey@0"
I20260928 21:20:52.339403 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "21acf65af8504fe8b67bff199b1a1cb3"
format_stamp: "Formatted at 2026-09-28 21:20:52 on dist-test-slave-rc9f"
server_key: "fc7358813923e1e9681c656d44b8255f"
server_key_iv: "10c18bcde8e2dab5398040549b6de4df"
server_key_version: "encryptionkey@0"
I20260928 21:20:52.343318 25445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:20:52.346187 25460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:52.347527 25445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:52.347678 25445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "21acf65af8504fe8b67bff199b1a1cb3"
format_stamp: "Formatted at 2026-09-28 21:20:52 on dist-test-slave-rc9f"
server_key: "fc7358813923e1e9681c656d44b8255f"
server_key_iv: "10c18bcde8e2dab5398040549b6de4df"
server_key_version: "encryptionkey@0"
I20260928 21:20:52.347803 25445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:52.375200 25445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:52.378439 25445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:52.378628 25445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:52.385673 25445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41199
I20260928 21:20:52.385673 25512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41199 every 8 connection(s)
I20260928 21:20:52.386706 25445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:52.389526 25513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:52.393800 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25445
I20260928 21:20:52.393918 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:52.394171 5195 external_mini_cluster.cc:1428] Setting key d65972ab1309cbc342364f476e920f75
I20260928 21:20:52.395416 25513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3: Bootstrap starting.
I20260928 21:20:52.398363 25513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:52.399304 25513 log.cc:824] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:52.401610 25513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3: No bootstrap required, opened a new log
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:20:52.406086 25513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } }
I20260928 21:20:52.406385 25513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:52.406451 25513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21acf65af8504fe8b67bff199b1a1cb3, State: Initialized, Role: FOLLOWER
I20260928 21:20:52.406962 25513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } }
I20260928 21:20:52.407106 25513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:52.407177 25513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:52.407281 25513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:52.408464 25513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } }
I20260928 21:20:52.408851 25513 leader_election.cc:302] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21acf65af8504fe8b67bff199b1a1cb3; no voters:
I20260928 21:20:52.409190 25513 leader_election.cc:288] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:52.409309 25518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:52.409529 25518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [term 1 LEADER]: Becoming Leader. State: Replica: 21acf65af8504fe8b67bff199b1a1cb3, State: Running, Role: LEADER
I20260928 21:20:52.409873 25518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } }
I20260928 21:20:52.409993 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:52.395628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56409 (local address 127.0.0.1:41199)
0928 21:20:52.396055 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:52.396075 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:52.396118 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:20:52.397014 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:52.397039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:52.397549 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:52.397986 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:52.398756 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:52.400082 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:52.401007 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:52.401745 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:52.401920 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:52.404629 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:52.404637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:52.404658 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:52.404693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:52.406726 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:52.407216 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:52.407222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:52.407228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:52.407385 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:52.407688 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:52.407691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:52.407693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:52.408217 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:52.408431 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:52.408694 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 21:20:52.408923 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260928 21:20:52.410517 25513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:52.411022 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21acf65af8504fe8b67bff199b1a1cb3. Latest consensus state: current_term: 1 leader_uuid: "21acf65af8504fe8b67bff199b1a1cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } } }
I20260928 21:20:52.411134 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:52.411407 25520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21acf65af8504fe8b67bff199b1a1cb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21acf65af8504fe8b67bff199b1a1cb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41199 } } }
I20260928 21:20:52.411481 25520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:52.411514 25523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:52.413512 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:52.414783 25523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:52.422323 25523 catalog_manager.cc:1347] Generated new cluster ID: 4ed89808ca274977a2ab27480d5ccd4c
I20260928 21:20:52.422410 25523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:52.424082 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:52.415824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35715 (local address 127.0.0.1:41199)
0928 21:20:52.415946 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:52.415951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:52.416373 (+ 422us) server_negotiation.cc:407] Connection header received
0928 21:20:52.416531 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:52.416534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:52.416587 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:52.416676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:52.417436 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:52.418022 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:52.418819 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:52.419058 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:52.419154 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:52.420660 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:52.420664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:52.420667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:52.420693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:52.422333 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:52.422868 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:52.422872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:52.422874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:52.422950 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:52.423251 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:52.423254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:52.423256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:52.423440 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:52.423565 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:52.423824 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 21:20:52.423917 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:20:52.436916 25523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:52.438118 25523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:52.445717 25523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21acf65af8504fe8b67bff199b1a1cb3: Generated new TSK 0
I20260928 21:20:52.446451 25523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:52.456390 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25445
W20260928 21:20:52.461150 25430 connection.cc:535] client connection to 127.0.0.1:41199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41199): Connection reset by peer (error 104)
2026-09-28T21:20:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25544](info): set up 2 sockets
Sep 28 21:20:52 dist-test-slave-rc9f krb5kdc[25544](info): commencing operation
krb5kdc: starting...
W20260928 21:20:53.417913 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:53 dist-test-slave-rc9f krb5kdc[25544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:53.535895 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:53.647061 25557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:53.647372 25557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:53.647432 25557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:53.651616 25557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:53.651700 25557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:53.651727 25557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:53.651750 25557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:53.651779 25557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:53.657410 25557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:53.658596 25557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:53.659644 25557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:53.668258 25563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:53.668300 25557 server_base.cc:1034] running on GCE node
W20260928 21:20:53.668288 25565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:53.668244 25562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:53.669099 25557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:53.670313 25557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:53.671494 25557 hybrid_clock.cc:648] HybridClock initialized: now 1790630453671463 us; error 44 us; skew 500 ppm
Sep 28 21:20:53 dist-test-slave-rc9f krb5kdc[25544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:53.679443 25557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:53.680529 25557 webserver.cc:460] Webserver started at http://127.0.0.1:41881/ using document root <none> and password file <none>
I20260928 21:20:53.681108 25557 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:53.681193 25557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:53.681417 25557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:53.683617 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "74f25878d091476497268e2dbb6cd61f"
format_stamp: "Formatted at 2026-09-28 21:20:53 on dist-test-slave-rc9f"
server_key: "6bb4a4b00d0070612dec2d5abf9fce58"
server_key_iv: "8c3b44c9e852f52cf631a28629324193"
server_key_version: "encryptionkey@0"
I20260928 21:20:53.684197 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "74f25878d091476497268e2dbb6cd61f"
format_stamp: "Formatted at 2026-09-28 21:20:53 on dist-test-slave-rc9f"
server_key: "6bb4a4b00d0070612dec2d5abf9fce58"
server_key_iv: "8c3b44c9e852f52cf631a28629324193"
server_key_version: "encryptionkey@0"
I20260928 21:20:53.688068 25557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:20:53.690630 25572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:53.691872 25557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:20:53.692003 25557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "74f25878d091476497268e2dbb6cd61f"
format_stamp: "Formatted at 2026-09-28 21:20:53 on dist-test-slave-rc9f"
server_key: "6bb4a4b00d0070612dec2d5abf9fce58"
server_key_iv: "8c3b44c9e852f52cf631a28629324193"
server_key_version: "encryptionkey@0"
I20260928 21:20:53.692114 25557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:53.707331 25557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:53.710572 25557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:53.710767 25557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:53.717823 25557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40547
I20260928 21:20:53.717844 25624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40547 every 8 connection(s)
I20260928 21:20:53.718879 25557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:53.721686 25625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:53.722298 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25557
I20260928 21:20:53.722419 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:53.722708 5195 external_mini_cluster.cc:1428] Setting key 419e8e9a272a5a4b07c6077095b5e472
I20260928 21:20:53.728817 25625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f: Bootstrap starting.
I20260928 21:20:53.731166 25625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:53.731202 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:53.723924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48647 (local address 127.0.0.1:40547)
0928 21:20:53.724328 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:53.724343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:53.724379 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:20:53.725075 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:53.725095 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:53.725456 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:53.725829 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:53.726238 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:53.727222 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:53.728085 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:53.728567 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:53.728705 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:53.729599 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:53.729606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:53.729623 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:53.730001 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:53.730055 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:53.730079 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:53.730201 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:53.730357 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:20:53.730489 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":112,"threads_started":1}
I20260928 21:20:53.731907 25625 log.cc:824] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:53.733983 25625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f: No bootstrap required, opened a new log
I20260928 21:20:53.734019 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:53.737835 25625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } }
I20260928 21:20:53.738090 25625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:53.738161 25625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f25878d091476497268e2dbb6cd61f, State: Initialized, Role: FOLLOWER
I20260928 21:20:53.738579 25625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } }
I20260928 21:20:53.738710 25625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:53.738776 25625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:53.738896 25625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:53.739770 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:53.735913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60889 (local address 127.0.0.1:40547)
0928 21:20:53.736020 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:53.736026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:53.736219 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:20:53.736326 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:53.736329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:53.736370 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:53.736433 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:53.736722 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:53.737226 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:53.738079 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:53.738393 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:53.738487 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:53.739330 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:53.739333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:53.739336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:53.739349 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:53.739373 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:53.739383 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:53.739450 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:53.739593 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:20:53.739628 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:20:53.739971 25625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } }
I20260928 21:20:53.740319 25625 leader_election.cc:302] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f25878d091476497268e2dbb6cd61f; no voters:
I20260928 21:20:53.740644 25625 leader_election.cc:288] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:53.740736 25635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:53.740949 25635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [term 1 LEADER]: Becoming Leader. State: Replica: 74f25878d091476497268e2dbb6cd61f, State: Running, Role: LEADER
I20260928 21:20:53.741240 25635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } }
I20260928 21:20:53.741930 25625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:53.742883 25637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f25878d091476497268e2dbb6cd61f. Latest consensus state: current_term: 1 leader_uuid: "74f25878d091476497268e2dbb6cd61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } } }
I20260928 21:20:53.742988 25637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:53.742858 25636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f25878d091476497268e2dbb6cd61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f25878d091476497268e2dbb6cd61f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40547 } } }
I20260928 21:20:53.743201 25636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:53.743539 25644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:53.746708 25644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:53.753495 25644 catalog_manager.cc:1347] Generated new cluster ID: 137a462f717540a6a7acf367e867053f
I20260928 21:20:53.753612 25644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:53.772791 25644 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:53.773828 25644 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:53.780637 25644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f25878d091476497268e2dbb6cd61f: Generated new TSK 0
I20260928 21:20:53.781416 25644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:53.795979 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25557
W20260928 21:20:53.800841 25542 connection.cc:535] client connection to 127.0.0.1:40547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40547): Connection reset by peer (error 104)
2026-09-28T21:20:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:53 dist-test-slave-rc9f krb5kdc[25656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:53 dist-test-slave-rc9f krb5kdc[25656](info): set up 2 sockets
Sep 28 21:20:53 dist-test-slave-rc9f krb5kdc[25656](info): commencing operation
krb5kdc: starting...
W20260928 21:20:54.745457 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:54 dist-test-slave-rc9f krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:54.866783 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:54.981503 25669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:54.981851 25669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:54.981915 25669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:54.986110 25669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:54.986196 25669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:54.986224 25669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:54.986244 25669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:54.986265 25669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:54.992202 25669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:54.993386 25669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:54.994516 25669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:55.003382 25674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:55.003454 25669 server_base.cc:1034] running on GCE node
W20260928 21:20:55.003381 25677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:55.003381 25675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:55.004328 25669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:55.005502 25669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:55.006708 25669 hybrid_clock.cc:648] HybridClock initialized: now 1790630455006687 us; error 38 us; skew 500 ppm
Sep 28 21:20:55 dist-test-slave-rc9f krb5kdc[25656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:55.009516 25669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:55.010586 25669 webserver.cc:460] Webserver started at http://127.0.0.1:40101/ using document root <none> and password file <none>
I20260928 21:20:55.011373 25669 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:55.011466 25669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:55.011695 25669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:55.013983 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2"
format_stamp: "Formatted at 2026-09-28 21:20:55 on dist-test-slave-rc9f"
server_key: "610650346663eb13444eb91e38655f87"
server_key_iv: "1ddd897b1258fe170690cc1d543b9b06"
server_key_version: "encryptionkey@0"
I20260928 21:20:55.014613 25669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2"
format_stamp: "Formatted at 2026-09-28 21:20:55 on dist-test-slave-rc9f"
server_key: "610650346663eb13444eb91e38655f87"
server_key_iv: "1ddd897b1258fe170690cc1d543b9b06"
server_key_version: "encryptionkey@0"
I20260928 21:20:55.018705 25669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:55.021412 25684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:55.022795 25669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:55.022958 25669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2"
format_stamp: "Formatted at 2026-09-28 21:20:55 on dist-test-slave-rc9f"
server_key: "610650346663eb13444eb91e38655f87"
server_key_iv: "1ddd897b1258fe170690cc1d543b9b06"
server_key_version: "encryptionkey@0"
I20260928 21:20:55.023085 25669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:55.028072 25669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:55.031423 25669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:55.031615 25669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:55.038988 25669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34787
I20260928 21:20:55.039006 25736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34787 every 8 connection(s)
I20260928 21:20:55.040052 25669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:55.043211 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25669
I20260928 21:20:55.043252 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:55.043361 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:55.043629 5195 external_mini_cluster.cc:1428] Setting key 4b2c7a1e4c49c1396e649334124f75ad
I20260928 21:20:55.049525 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:55.045150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33215 (local address 127.0.0.1:34787)
0928 21:20:55.045673 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:55.045693 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:55.045740 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:20:55.046482 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:55.046506 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:55.046912 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:55.047332 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:55.048005 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:55.048011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:55.048027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:55.048039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:55.048102 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:55.048129 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:55.048288 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:55.048355 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 21:20:55.048533 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260928 21:20:55.051120 25737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2: Bootstrap starting.
I20260928 21:20:55.052579 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:55.053637 25737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:55.054445 25737 log.cc:824] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:55.056289 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:55.054580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49921 (local address 127.0.0.1:34787)
0928 21:20:55.054705 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:55.054711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:55.054894 (+ 183us) server_negotiation.cc:407] Connection header received
0928 21:20:55.055005 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:55.055009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:55.055045 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:55.055128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:55.055913 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:55.055916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:55.055918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:55.055929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:55.055956 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:55.055966 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:55.056033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:55.056126 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:20:55.056159 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:20:55.056941 25737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2: No bootstrap required, opened a new log
I20260928 21:20:55.060580 25737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } }
I20260928 21:20:55.060863 25737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:55.060927 25737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d28a64fc87e4b2ca38a89e55ce3a8e2, State: Initialized, Role: FOLLOWER
I20260928 21:20:55.061386 25737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } }
I20260928 21:20:55.061523 25737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:55.061607 25737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:55.061720 25737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:55.062937 25737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } }
I20260928 21:20:55.063356 25737 leader_election.cc:302] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d28a64fc87e4b2ca38a89e55ce3a8e2; no voters:
I20260928 21:20:55.063755 25737 leader_election.cc:288] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:55.063849 25747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:55.064067 25747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 4d28a64fc87e4b2ca38a89e55ce3a8e2, State: Running, Role: LEADER
I20260928 21:20:55.064410 25747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } }
I20260928 21:20:55.065258 25737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:55.066396 25749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d28a64fc87e4b2ca38a89e55ce3a8e2. Latest consensus state: current_term: 1 leader_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } } }
I20260928 21:20:55.066546 25749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:55.066380 25748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d28a64fc87e4b2ca38a89e55ce3a8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34787 } } }
I20260928 21:20:55.066766 25748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:55.066918 25756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:55.070451 25756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:55.077255 25756 catalog_manager.cc:1347] Generated new cluster ID: cfc9aa7903d44abfaa34d3f4073dbf0b
I20260928 21:20:55.077348 25756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:55.096990 25756 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:55.097357 25756 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:55.102411 25756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d28a64fc87e4b2ca38a89e55ce3a8e2: Generated new TSK 0
I20260928 21:20:55.103143 25756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:55.111239 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25669
2026-09-28T21:20:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:55 dist-test-slave-rc9f krb5kdc[25768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:55 dist-test-slave-rc9f krb5kdc[25768](info): set up 2 sockets
Sep 28 21:20:55 dist-test-slave-rc9f krb5kdc[25768](info): commencing operation
krb5kdc: starting...
W20260928 21:20:56.065475 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:56.189986 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:56.304741 25781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:56.305063 25781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:56.305106 25781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:56.309360 25781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:56.309458 25781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:56.309477 25781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:56.309494 25781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:56.309507 25781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:56.315351 25781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:56.316524 25781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:56.317550 25781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:20:56.326509 25781 server_base.cc:1034] running on GCE node
W20260928 21:20:56.326464 25787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:56.326464 25786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:56.326476 25789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:56.327358 25781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:56.328564 25781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:56.329850 25781 hybrid_clock.cc:648] HybridClock initialized: now 1790630456329775 us; error 93 us; skew 500 ppm
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:56.332957 25781 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:56.334043 25781 webserver.cc:460] Webserver started at http://127.0.0.1:40757/ using document root <none> and password file <none>
I20260928 21:20:56.334666 25781 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:56.334743 25781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:56.334983 25781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:56.337260 25781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "13beaa8c1ce5430882f80bf9cd79a5b5"
format_stamp: "Formatted at 2026-09-28 21:20:56 on dist-test-slave-rc9f"
server_key: "b97862a16813e8a3bcecf41a560a9e15"
server_key_iv: "32c324bb21c8f4418a6bd4831fcac370"
server_key_version: "encryptionkey@0"
I20260928 21:20:56.337906 25781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "13beaa8c1ce5430882f80bf9cd79a5b5"
format_stamp: "Formatted at 2026-09-28 21:20:56 on dist-test-slave-rc9f"
server_key: "b97862a16813e8a3bcecf41a560a9e15"
server_key_iv: "32c324bb21c8f4418a6bd4831fcac370"
server_key_version: "encryptionkey@0"
I20260928 21:20:56.342196 25781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:20:56.344893 25796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:56.346283 25781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:56.346417 25781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "13beaa8c1ce5430882f80bf9cd79a5b5"
format_stamp: "Formatted at 2026-09-28 21:20:56 on dist-test-slave-rc9f"
server_key: "b97862a16813e8a3bcecf41a560a9e15"
server_key_iv: "32c324bb21c8f4418a6bd4831fcac370"
server_key_version: "encryptionkey@0"
I20260928 21:20:56.346535 25781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:56.360034 25781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:56.363507 25781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:56.363713 25781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:56.371536 25781 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44439
I20260928 21:20:56.371585 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44439 every 8 connection(s)
I20260928 21:20:56.372550 25781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:56.375546 25849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:56.377133 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25781
I20260928 21:20:56.377260 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:56.377529 5195 external_mini_cluster.cc:1428] Setting key 9352488b4239c28996c6de307c20b43f
I20260928 21:20:56.382212 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: Bootstrap starting.
I20260928 21:20:56.384625 25849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:56.385490 25849 log.cc:824] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:20:56.387854 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: No bootstrap required, opened a new log
I20260928 21:20:56.391603 25849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } }
I20260928 21:20:56.391842 25849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:56.391878 25849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13beaa8c1ce5430882f80bf9cd79a5b5, State: Initialized, Role: FOLLOWER
I20260928 21:20:56.392274 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:56.379056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57459 (local address 127.0.0.1:44439)
0928 21:20:56.379583 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:56.379609 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:56.379659 (+ 50us) server_negotiation.cc:407] Connection header received
0928 21:20:56.380380 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:56.380400 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:56.380795 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:56.381197 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:56.382088 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:56.383147 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:56.384045 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:56.384545 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:56.384697 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:56.387542 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:56.387547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:56.387562 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:56.387589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:56.389317 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:56.389836 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:56.389841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:56.389846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:56.389964 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:56.390230 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:56.390233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:56.390234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:56.390652 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:56.390852 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:56.391147 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:20:56.391440 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":143,"threads_started":1}
I20260928 21:20:56.392331 25849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } }
I20260928 21:20:56.392482 25849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:56.392544 25849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:56.392640 25849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:56.393595 25849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } }
I20260928 21:20:56.393937 25849 leader_election.cc:302] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13beaa8c1ce5430882f80bf9cd79a5b5; no voters:
I20260928 21:20:56.394263 25849 leader_election.cc:288] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:56.394366 25854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:56.394596 25854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [term 1 LEADER]: Becoming Leader. State: Replica: 13beaa8c1ce5430882f80bf9cd79a5b5, State: Running, Role: LEADER
I20260928 21:20:56.394954 25854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } }
I20260928 21:20:56.395339 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:56.395654 25849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:56.396102 25854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13beaa8c1ce5430882f80bf9cd79a5b5. Latest consensus state: current_term: 1 leader_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } } }
I20260928 21:20:56.396757 25854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:56.396651 25855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13beaa8c1ce5430882f80bf9cd79a5b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44439 } } }
I20260928 21:20:56.397010 25855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5 [sys.catalog]: This master's current role is: LEADER
W20260928 21:20:56.401080 25873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:20:56.401167 25873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:20:56.401232 25867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:56.402113 25867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:56.407037 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:56.398423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56881 (local address 127.0.0.1:44439)
0928 21:20:56.398532 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:56.398536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:56.398707 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:20:56.398837 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:56.398840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:56.398889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:56.398976 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:56.399685 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:56.400437 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:56.401557 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:56.401927 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:56.402039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:56.403692 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:56.403696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:56.403699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:20:56.403726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:56.405081 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:56.405499 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:56.405503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:56.405505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:56.405621 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:20:56.405975 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:20:56.405979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:20:56.405981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:20:56.406229 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:56.406332 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:56.406733 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 21:20:56.406875 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:20:56.408758 25867 catalog_manager.cc:1347] Generated new cluster ID: 346d4819fa4d4ebd919ac1b8fb81a016
I20260928 21:20:56.408895 25867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:56.422947 25867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:56.424417 25867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:56.431483 25867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13beaa8c1ce5430882f80bf9cd79a5b5: Generated new TSK 0
I20260928 21:20:56.432428 25867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:56.436244 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25781
2026-09-28T21:20:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25880](info): set up 2 sockets
Sep 28 21:20:56 dist-test-slave-rc9f krb5kdc[25880](info): commencing operation
krb5kdc: starting...
W20260928 21:20:57.397446 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:57 dist-test-slave-rc9f krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:57.516223 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:57.626407 25893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:57.626693 25893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:57.626734 25893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:57.630779 25893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:57.630852 25893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:57.630872 25893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:57.630887 25893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:57.630903 25893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:57.636502 25893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:57.637632 25893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:57.638667 25893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:57.647135 25898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:57.647135 25899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:57.647383 25893 server_base.cc:1034] running on GCE node
W20260928 21:20:57.647459 25901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:57.648025 25893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:57.649134 25893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:57.650298 25893 hybrid_clock.cc:648] HybridClock initialized: now 1790630457650282 us; error 30 us; skew 500 ppm
Sep 28 21:20:57 dist-test-slave-rc9f krb5kdc[25880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:57.652941 25893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:57.653873 25893 webserver.cc:460] Webserver started at http://127.0.0.1:41975/ using document root <none> and password file <none>
I20260928 21:20:57.654418 25893 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:57.654490 25893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:57.654696 25893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:57.656711 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "2370c5d1f1d6413b829973fc383455bb"
format_stamp: "Formatted at 2026-09-28 21:20:57 on dist-test-slave-rc9f"
server_key: "45bfc64dc94244af658667feed7acd62"
server_key_iv: "0fdd715a633e6f8ec58f48940c7fcc9a"
server_key_version: "encryptionkey@0"
I20260928 21:20:57.657258 25893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "2370c5d1f1d6413b829973fc383455bb"
format_stamp: "Formatted at 2026-09-28 21:20:57 on dist-test-slave-rc9f"
server_key: "45bfc64dc94244af658667feed7acd62"
server_key_iv: "0fdd715a633e6f8ec58f48940c7fcc9a"
server_key_version: "encryptionkey@0"
I20260928 21:20:57.661039 25893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:20:57.663570 25908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:57.664914 25893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:20:57.665040 25893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "2370c5d1f1d6413b829973fc383455bb"
format_stamp: "Formatted at 2026-09-28 21:20:57 on dist-test-slave-rc9f"
server_key: "45bfc64dc94244af658667feed7acd62"
server_key_iv: "0fdd715a633e6f8ec58f48940c7fcc9a"
server_key_version: "encryptionkey@0"
I20260928 21:20:57.665153 25893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:57.684238 25893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:57.687403 25893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:57.687595 25893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:57.694538 25893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35555
I20260928 21:20:57.694674 25960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35555 every 8 connection(s)
I20260928 21:20:57.695533 25893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:57.698542 25961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:57.702821 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 25893
I20260928 21:20:57.702975 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:57.703222 5195 external_mini_cluster.cc:1428] Setting key 6f95ec67e3686e854fac4dd4c750e748
I20260928 21:20:57.704921 25961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb: Bootstrap starting.
I20260928 21:20:57.707382 25961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:57.708150 25961 log.cc:824] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:57.710284 25961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb: No bootstrap required, opened a new log
I20260928 21:20:57.712343 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:57.704634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53423 (local address 127.0.0.1:35555)
0928 21:20:57.705061 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:57.705082 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:57.705124 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:20:57.705849 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:57.705869 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:57.706221 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:57.706515 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:57.706857 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:57.707897 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:57.708953 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:57.709426 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:57.709555 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:57.710441 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:57.710446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:57.710460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:57.710857 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:57.710912 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:57.710936 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:57.711012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:57.711160 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 21:20:57.711369 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260928 21:20:57.713457 25961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } }
I20260928 21:20:57.713752 25961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:57.713815 25961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2370c5d1f1d6413b829973fc383455bb, State: Initialized, Role: FOLLOWER
I20260928 21:20:57.714265 25961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } }
I20260928 21:20:57.714423 25961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:57.714486 25961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:57.714574 25961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:57.715284 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:57.715553 25961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } }
I20260928 21:20:57.715859 25961 leader_election.cc:302] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2370c5d1f1d6413b829973fc383455bb; no voters:
I20260928 21:20:57.716140 25961 leader_election.cc:288] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:57.716220 25966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:57.716421 25966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [term 1 LEADER]: Becoming Leader. State: Replica: 2370c5d1f1d6413b829973fc383455bb, State: Running, Role: LEADER
I20260928 21:20:57.716730 25966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } }
I20260928 21:20:57.717461 25961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:57.717931 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2370c5d1f1d6413b829973fc383455bb. Latest consensus state: current_term: 1 leader_uuid: "2370c5d1f1d6413b829973fc383455bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } } }
I20260928 21:20:57.718041 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:57.718222 25966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2370c5d1f1d6413b829973fc383455bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2370c5d1f1d6413b829973fc383455bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35555 } } }
I20260928 21:20:57.718307 25966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:57.718560 25975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:57.721906 25975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:57.721958 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:57.717230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41473 (local address 127.0.0.1:35555)
0928 21:20:57.717336 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:57.717341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:57.717699 (+ 358us) server_negotiation.cc:407] Connection header received
0928 21:20:57.717734 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:57.717736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:57.717781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:57.717847 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:57.718216 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:57.718956 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:57.720044 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:20:57.720375 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:20:57.720465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:20:57.721337 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:57.721341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:57.721344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:57.721358 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:57.721388 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:57.721401 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:57.721460 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:57.721602 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:20:57.721676 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:20:57.728094 25975 catalog_manager.cc:1347] Generated new cluster ID: 3c21ececd85f49288146936fd7a918d0
I20260928 21:20:57.728195 25975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:57.743239 25975 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:57.744189 25975 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:57.748157 25975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2370c5d1f1d6413b829973fc383455bb: Generated new TSK 0
I20260928 21:20:57.748807 25975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:57.758724 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 25893
2026-09-28T21:20:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:57 dist-test-slave-rc9f krb5kdc[25992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:57 dist-test-slave-rc9f krb5kdc[25992](info): set up 2 sockets
Sep 28 21:20:57 dist-test-slave-rc9f krb5kdc[25992](info): commencing operation
krb5kdc: starting...
W20260928 21:20:58.717465 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:20:58 dist-test-slave-rc9f krb5kdc[25992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:20:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:20:58.838299 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:20:58.952035 26005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:20:58.952329 26005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:20:58.952376 26005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:20:58.956700 26005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:20:58.956779 26005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:20:58.956797 26005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:20:58.956813 26005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:20:58.956827 26005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:20:58.962395 26005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:20:58.963513 26005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:20:58.964556 26005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:20:58.973232 26010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:58.973249 26013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:20:58.973244 26011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:20:58.973301 26005 server_base.cc:1034] running on GCE node
I20260928 21:20:58.974074 26005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:20:58.975245 26005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:20:58.976431 26005 hybrid_clock.cc:648] HybridClock initialized: now 1790630458976417 us; error 30 us; skew 500 ppm
Sep 28 21:20:58 dist-test-slave-rc9f krb5kdc[25992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:20:58.981968 26005 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:20:58.982977 26005 webserver.cc:460] Webserver started at http://127.0.0.1:37689/ using document root <none> and password file <none>
I20260928 21:20:58.983547 26005 fs_manager.cc:359] Metadata directory not provided
I20260928 21:20:58.983618 26005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:20:58.983830 26005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:20:58.986045 26005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "9ec2176f5bf24449804485155d1d36b0"
format_stamp: "Formatted at 2026-09-28 21:20:58 on dist-test-slave-rc9f"
server_key: "03b98f6d549d5d725d12a8bbce0ae80c"
server_key_iv: "2fa8fffabcf84d1c3981409c0e2c78ae"
server_key_version: "encryptionkey@0"
I20260928 21:20:58.986622 26005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "9ec2176f5bf24449804485155d1d36b0"
format_stamp: "Formatted at 2026-09-28 21:20:58 on dist-test-slave-rc9f"
server_key: "03b98f6d549d5d725d12a8bbce0ae80c"
server_key_iv: "2fa8fffabcf84d1c3981409c0e2c78ae"
server_key_version: "encryptionkey@0"
I20260928 21:20:58.990407 26005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:20:58.992981 26020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:20:58.994364 26005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:20:58.994526 26005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "9ec2176f5bf24449804485155d1d36b0"
format_stamp: "Formatted at 2026-09-28 21:20:58 on dist-test-slave-rc9f"
server_key: "03b98f6d549d5d725d12a8bbce0ae80c"
server_key_iv: "2fa8fffabcf84d1c3981409c0e2c78ae"
server_key_version: "encryptionkey@0"
I20260928 21:20:58.994658 26005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:20:58.999640 26005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:20:59.002975 26005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:20:59.003168 26005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:20:59.009994 26005 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44149
I20260928 21:20:59.010038 26072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44149 every 8 connection(s)
I20260928 21:20:59.010963 26005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:20:59.013895 26073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:20:59.014401 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26005
I20260928 21:20:59.014530 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:20:59.014813 5195 external_mini_cluster.cc:1428] Setting key 2993a5477eb7775877388291e420c226
I20260928 21:20:59.020637 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:59.016280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54349 (local address 127.0.0.1:44149)
0928 21:20:59.016750 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:59.016775 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:59.016821 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:20:59.017666 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:59.017692 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:59.018087 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:59.018476 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:59.019190 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:59.019196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:59.019212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:59.019224 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:59.019286 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:59.019312 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:59.019449 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:59.019516 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 21:20:59.019676 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260928 21:20:59.021128 26073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0: Bootstrap starting.
I20260928 21:20:59.023792 26073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0: Neither blocks nor log segments found. Creating new log.
I20260928 21:20:59.023874 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:20:59.024685 26073 log.cc:824] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:20:59.027042 26073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0: No bootstrap required, opened a new log
I20260928 21:20:59.027621 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:20:59.026076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58275 (local address 127.0.0.1:44149)
0928 21:20:59.026209 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:20:59.026213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:20:59.026377 (+ 164us) server_negotiation.cc:407] Connection header received
0928 21:20:59.026496 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:20:59.026499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:20:59.026530 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:20:59.026593 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:20:59.027267 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:20:59.027269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:20:59.027271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:20:59.027281 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:20:59.027306 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:20:59.027316 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:20:59.027383 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:20:59.027473 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:20:59.027504 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:20:59.030987 26073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 21:20:59.031270 26073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:20:59.031332 26073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec2176f5bf24449804485155d1d36b0, State: Initialized, Role: FOLLOWER
I20260928 21:20:59.031888 26073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 21:20:59.032064 26073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:20:59.032135 26073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:20:59.032239 26073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:20:59.033279 26073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 21:20:59.033701 26073 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec2176f5bf24449804485155d1d36b0; no voters:
I20260928 21:20:59.034039 26073 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:20:59.034159 26083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:20:59.034364 26083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [term 1 LEADER]: Becoming Leader. State: Replica: 9ec2176f5bf24449804485155d1d36b0, State: Running, Role: LEADER
I20260928 21:20:59.034708 26083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260928 21:20:59.035315 26073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:20:59.036401 26085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec2176f5bf24449804485155d1d36b0. Latest consensus state: current_term: 1 leader_uuid: "9ec2176f5bf24449804485155d1d36b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260928 21:20:59.036553 26085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:59.036401 26084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec2176f5bf24449804485155d1d36b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec2176f5bf24449804485155d1d36b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260928 21:20:59.036680 26084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:20:59.037075 26092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:20:59.040371 26092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:20:59.046720 26092 catalog_manager.cc:1347] Generated new cluster ID: 5d7cacb8337c4de595debc318877ad6b
I20260928 21:20:59.046821 26092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:20:59.074735 26092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:20:59.075125 26092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:20:59.081668 26092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec2176f5bf24449804485155d1d36b0: Generated new TSK 0
I20260928 21:20:59.082449 26092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:20:59.094844 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26005
2026-09-28T21:20:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:20:59 dist-test-slave-rc9f krb5kdc[26104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:20:59 dist-test-slave-rc9f krb5kdc[26104](info): set up 2 sockets
Sep 28 21:20:59 dist-test-slave-rc9f krb5kdc[26104](info): commencing operation
krb5kdc: starting...
W20260928 21:21:00.053684 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:00.172703 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:00.285159 26117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:00.285476 26117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:00.285518 26117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:00.289705 26117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:00.289788 26117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:00.289806 26117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:00.289824 26117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:00.289837 26117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:00.295408 26117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:00.296576 26117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:00.297657 26117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:00.306438 26117 server_base.cc:1034] running on GCE node
W20260928 21:21:00.306366 26123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:00.306371 26122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:00.306663 26125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:00.307264 26117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:00.308516 26117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:00.309638 26117 hybrid_clock.cc:648] HybridClock initialized: now 1790630460309607 us; error 51 us; skew 500 ppm
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:00.312456 26117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:00.313416 26117 webserver.cc:460] Webserver started at http://127.0.0.1:41003/ using document root <none> and password file <none>
I20260928 21:21:00.313997 26117 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:00.314066 26117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:00.314282 26117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:00.316370 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "86ca8b28d375486e8a59ff709b4b9a1c"
format_stamp: "Formatted at 2026-09-28 21:21:00 on dist-test-slave-rc9f"
server_key: "26c8943f12ac13ef76bf6c1c8fa8c439"
server_key_iv: "ad476a553b001ff05695e4591298bb3e"
server_key_version: "encryptionkey@0"
I20260928 21:21:00.316908 26117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "86ca8b28d375486e8a59ff709b4b9a1c"
format_stamp: "Formatted at 2026-09-28 21:21:00 on dist-test-slave-rc9f"
server_key: "26c8943f12ac13ef76bf6c1c8fa8c439"
server_key_iv: "ad476a553b001ff05695e4591298bb3e"
server_key_version: "encryptionkey@0"
I20260928 21:21:00.320644 26117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:21:00.323179 26132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:00.324467 26117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:00.324600 26117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "86ca8b28d375486e8a59ff709b4b9a1c"
format_stamp: "Formatted at 2026-09-28 21:21:00 on dist-test-slave-rc9f"
server_key: "26c8943f12ac13ef76bf6c1c8fa8c439"
server_key_iv: "ad476a553b001ff05695e4591298bb3e"
server_key_version: "encryptionkey@0"
I20260928 21:21:00.324715 26117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:00.337554 26117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:00.340603 26117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:00.340797 26117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:00.347685 26117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44799
I20260928 21:21:00.347702 26184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44799 every 8 connection(s)
I20260928 21:21:00.348673 26117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:00.348912 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26117
I20260928 21:21:00.349030 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:00.349321 5195 external_mini_cluster.cc:1428] Setting key 0ce2be15388639c55c954636a582ee13
I20260928 21:21:00.351887 26185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:00.358726 26185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c: Bootstrap starting.
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:21:00.361516 26185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:00.362432 26185 log.cc:824] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:00.364704 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:00.350868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:35257 (local address 127.0.0.1:44799)
0928 21:21:00.351315 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:00.351330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:00.351367 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:21:00.352025 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:00.352049 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:00.352438 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:00.352819 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:00.353672 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:00.354724 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:00.355924 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:00.356425 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:00.356559 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:00.359485 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:00.359492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:00.359510 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:00.359547 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:00.361465 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:00.362038 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:00.362044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:00.362050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:00.362189 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:00.362458 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:00.362461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:00.362463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:00.363031 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:00.363187 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:00.363550 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 21:21:00.363783 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":145,"threads_started":1}
I20260928 21:21:00.364753 26185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c: No bootstrap required, opened a new log
I20260928 21:21:00.367619 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:00.368701 26185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260928 21:21:00.368976 26185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:00.369038 26185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ca8b28d375486e8a59ff709b4b9a1c, State: Initialized, Role: FOLLOWER
I20260928 21:21:00.369519 26185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260928 21:21:00.369743 26185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:00.369814 26185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:00.369916 26185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:00.370831 26185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260928 21:21:00.371152 26185 leader_election.cc:302] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86ca8b28d375486e8a59ff709b4b9a1c; no voters:
I20260928 21:21:00.371438 26185 leader_election.cc:288] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:00.371520 26195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:00.371699 26195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [term 1 LEADER]: Becoming Leader. State: Replica: 86ca8b28d375486e8a59ff709b4b9a1c, State: Running, Role: LEADER
I20260928 21:21:00.372073 26195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } }
I20260928 21:21:00.372766 26185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:00.373770 26196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260928 21:21:00.373898 26196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:00.373986 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86ca8b28d375486e8a59ff709b4b9a1c. Latest consensus state: current_term: 1 leader_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86ca8b28d375486e8a59ff709b4b9a1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44799 } } }
I20260928 21:21:00.374251 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:00.374411 26204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:00.377749 26204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:00.377787 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:00.369511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:56623 (local address 127.0.0.1:44799)
0928 21:21:00.369661 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:00.369665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:00.369821 (+ 156us) server_negotiation.cc:407] Connection header received
0928 21:21:00.369944 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:00.369947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:00.369997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:00.370062 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:00.370672 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:00.371251 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:00.372196 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:00.372433 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:00.372517 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:00.374121 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:00.374125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:00.374128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:00.374155 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:00.375877 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:00.376481 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:00.376485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:00.376486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:00.376572 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:00.376920 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:00.376923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:00.376925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:00.377104 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:00.377228 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:00.377508 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 21:21:00.377623 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:21:00.384747 26204 catalog_manager.cc:1347] Generated new cluster ID: ee8edecfec34475089f6df4daf39dc17
I20260928 21:21:00.384869 26204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:00.393890 26204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:00.395238 26204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:00.408277 26204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86ca8b28d375486e8a59ff709b4b9a1c: Generated new TSK 0
I20260928 21:21:00.409193 26204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:00.417910 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26117
W20260928 21:21:00.422662 26102 connection.cc:535] client connection to 127.0.0.1:44799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44799): Connection reset by peer (error 104)
2026-09-28T21:21:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26216](info): set up 2 sockets
Sep 28 21:21:00 dist-test-slave-rc9f krb5kdc[26216](info): commencing operation
krb5kdc: starting...
W20260928 21:21:01.373528 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:01 dist-test-slave-rc9f krb5kdc[26216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:01.501286 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:01.613853 26229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:01.614149 26229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:01.614198 26229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:01.618317 26229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:01.618391 26229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:01.618409 26229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:01.618425 26229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:01.618439 26229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:01.623940 26229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:01.625028 26229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:01.626060 26229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:01.635016 26234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:01.635016 26237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:01.635048 26235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:01.635100 26229 server_base.cc:1034] running on GCE node
I20260928 21:21:01.635838 26229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:01.637142 26229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:01.638330 26229 hybrid_clock.cc:648] HybridClock initialized: now 1790630461638316 us; error 39 us; skew 500 ppm
Sep 28 21:21:01 dist-test-slave-rc9f krb5kdc[26216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:01.641168 26229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:01.642163 26229 webserver.cc:460] Webserver started at http://127.0.0.1:34809/ using document root <none> and password file <none>
I20260928 21:21:01.642726 26229 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:01.642788 26229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:01.642976 26229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:01.645100 26229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "b520a87387b944709708b018144d205e"
format_stamp: "Formatted at 2026-09-28 21:21:01 on dist-test-slave-rc9f"
server_key: "3f9abff83efe7bc31507af02b562a85c"
server_key_iv: "eb01655dc0b974c4db9d28486f5ee559"
server_key_version: "encryptionkey@0"
I20260928 21:21:01.645714 26229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "b520a87387b944709708b018144d205e"
format_stamp: "Formatted at 2026-09-28 21:21:01 on dist-test-slave-rc9f"
server_key: "3f9abff83efe7bc31507af02b562a85c"
server_key_iv: "eb01655dc0b974c4db9d28486f5ee559"
server_key_version: "encryptionkey@0"
I20260928 21:21:01.649627 26229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:21:01.652330 26244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:01.653759 26229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:21:01.653908 26229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "b520a87387b944709708b018144d205e"
format_stamp: "Formatted at 2026-09-28 21:21:01 on dist-test-slave-rc9f"
server_key: "3f9abff83efe7bc31507af02b562a85c"
server_key_iv: "eb01655dc0b974c4db9d28486f5ee559"
server_key_version: "encryptionkey@0"
I20260928 21:21:01.654043 26229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:01.667783 26229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:01.671310 26229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:01.671514 26229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:01.678841 26229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34657
I20260928 21:21:01.678849 26296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34657 every 8 connection(s)
I20260928 21:21:01.679889 26229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:01.682844 26297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:01.687873 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26229
I20260928 21:21:01.688020 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:01.688279 5195 external_mini_cluster.cc:1428] Setting key 15b095d214d451e93f2d85289f488276
I20260928 21:21:01.689007 26297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e: Bootstrap starting.
I20260928 21:21:01.691565 26297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:01.692303 26297 log.cc:824] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:01.694269 26297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e: No bootstrap required, opened a new log
I20260928 21:21:01.697749 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:01.689863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:58689 (local address 127.0.0.1:34657)
0928 21:21:01.690349 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:01.690364 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:01.690402 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:21:01.691092 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:01.691110 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:01.691480 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:01.691839 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:01.692284 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:01.693317 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:01.694265 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:01.694810 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:01.694971 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:01.695941 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:01.695946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:01.695960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:01.696367 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:01.696434 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:01.696459 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:01.696575 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:01.696768 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:21:01.696913 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":131,"threads_started":1}
I20260928 21:21:01.698184 26297 raft_consensus.cc:348] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } }
I20260928 21:21:01.698472 26297 raft_consensus.cc:374] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:01.698521 26297 raft_consensus.cc:729] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b520a87387b944709708b018144d205e, State: Initialized, Role: FOLLOWER
I20260928 21:21:01.698975 26297 consensus_queue.cc:260] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } }
I20260928 21:21:01.699118 26297 raft_consensus.cc:388] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:01.699179 26297 raft_consensus.cc:482] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:01.699280 26297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:01.700372 26297 raft_consensus.cc:504] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } }
I20260928 21:21:01.700719 26297 leader_election.cc:302] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b520a87387b944709708b018144d205e; no voters:
I20260928 21:21:01.700915 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:01.701087 26297 leader_election.cc:288] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:01.701193 26302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:01.701448 26302 raft_consensus.cc:686] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [term 1 LEADER]: Becoming Leader. State: Replica: b520a87387b944709708b018144d205e, State: Running, Role: LEADER
I20260928 21:21:01.701846 26302 consensus_queue.cc:237] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } }
I20260928 21:21:01.702368 26297 sys_catalog.cc:564] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:01.703238 26302 sys_catalog.cc:455] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b520a87387b944709708b018144d205e. Latest consensus state: current_term: 1 leader_uuid: "b520a87387b944709708b018144d205e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } } }
I20260928 21:21:01.703430 26302 sys_catalog.cc:458] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:01.703626 26304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b520a87387b944709708b018144d205e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b520a87387b944709708b018144d205e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34657 } } }
I20260928 21:21:01.703727 26304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:01.704305 26316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:01.707858 26316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:01.710183 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:01.703417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:56361 (local address 127.0.0.1:34657)
0928 21:21:01.703524 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:01.703529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:01.704009 (+ 480us) server_negotiation.cc:407] Connection header received
0928 21:21:01.704641 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:01.704645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:01.704701 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:01.704749 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:01.706030 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:01.706790 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:01.707960 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:01.708283 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:01.708389 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:01.709315 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:01.709319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:01.709322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:01.709337 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:01.709367 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:01.709381 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:01.709463 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:01.710001 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 21:21:01.710040 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:21:01.713603 26316 catalog_manager.cc:1347] Generated new cluster ID: 89698d8e799341aa89f3fcec4b2ab4e9
I20260928 21:21:01.713676 26316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:01.719877 26316 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:01.721174 26316 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:01.735399 26316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b520a87387b944709708b018144d205e: Generated new TSK 0
I20260928 21:21:01.736348 26316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:01.744761 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26229
W20260928 21:21:01.749929 26214 connection.cc:535] client connection to 127.0.0.1:34657 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34657): Connection reset by peer (error 104)
2026-09-28T21:21:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:01 dist-test-slave-rc9f krb5kdc[26328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:01 dist-test-slave-rc9f krb5kdc[26328](info): set up 2 sockets
Sep 28 21:21:01 dist-test-slave-rc9f krb5kdc[26328](info): commencing operation
krb5kdc: starting...
W20260928 21:21:02.693418 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:02 dist-test-slave-rc9f krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:02.814145 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:02.922698 26341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:02.922996 26341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:02.923058 26341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:02.927135 26341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:02.927210 26341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:02.927238 26341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:02.927260 26341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:02.927286 26341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:02.932773 26341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:02.933986 26341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:02.935043 26341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:02.943656 26347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:02.943670 26349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:02.943634 26346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:02.944285 26341 server_base.cc:1034] running on GCE node
I20260928 21:21:02.944989 26341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:02.946488 26341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:02.947947 26341 hybrid_clock.cc:648] HybridClock initialized: now 1790630462947918 us; error 46 us; skew 500 ppm
Sep 28 21:21:02 dist-test-slave-rc9f krb5kdc[26328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:02.951535 26341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:02.952620 26341 webserver.cc:460] Webserver started at http://127.0.0.1:42897/ using document root <none> and password file <none>
I20260928 21:21:02.953193 26341 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:02.953262 26341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:02.953536 26341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:02.955662 26341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "04ae83ce99fd4dd0b2c9e7d74491894f"
format_stamp: "Formatted at 2026-09-28 21:21:02 on dist-test-slave-rc9f"
server_key: "d55f0498fb1289e5c65ce0fb1a8df32d"
server_key_iv: "7d0a4df67f0cec3883693446c038a99d"
server_key_version: "encryptionkey@0"
I20260928 21:21:02.956214 26341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "04ae83ce99fd4dd0b2c9e7d74491894f"
format_stamp: "Formatted at 2026-09-28 21:21:02 on dist-test-slave-rc9f"
server_key: "d55f0498fb1289e5c65ce0fb1a8df32d"
server_key_iv: "7d0a4df67f0cec3883693446c038a99d"
server_key_version: "encryptionkey@0"
I20260928 21:21:02.960243 26341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 21:21:02.962805 26356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:02.964107 26341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:02.964226 26341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "04ae83ce99fd4dd0b2c9e7d74491894f"
format_stamp: "Formatted at 2026-09-28 21:21:02 on dist-test-slave-rc9f"
server_key: "d55f0498fb1289e5c65ce0fb1a8df32d"
server_key_iv: "7d0a4df67f0cec3883693446c038a99d"
server_key_version: "encryptionkey@0"
I20260928 21:21:02.964336 26341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:02.969008 26341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:02.971948 26341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:02.972119 26341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:02.978693 26341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41941
I20260928 21:21:02.978704 26408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41941 every 8 connection(s)
I20260928 21:21:02.979599 26341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:02.979951 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26341
I20260928 21:21:02.980095 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:02.980360 5195 external_mini_cluster.cc:1428] Setting key ff752eb2d138a3cfec76cad130a7d907
I20260928 21:21:02.982838 26409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:02.985857 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:02.981752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:48599 (local address 127.0.0.1:41941)
0928 21:21:02.982146 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:02.982164 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:02.982206 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:21:02.983052 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:02.983073 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:02.983434 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:02.983782 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:02.984430 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:02.984435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:02.984449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:02.984460 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:02.984514 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:02.984538 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:02.984666 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:02.984729 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 21:21:02.984900 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260928 21:21:02.988451 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:02.989063 26409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f: Bootstrap starting.
I20260928 21:21:02.991474 26409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:02.991961 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:02.990382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:58317 (local address 127.0.0.1:41941)
0928 21:21:02.990487 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:02.990491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:02.990661 (+ 170us) server_negotiation.cc:407] Connection header received
0928 21:21:02.990769 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:02.990772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:02.990805 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:02.990886 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:02.991592 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:02.991596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:02.991599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:02.991609 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:02.991634 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:02.991644 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:02.991710 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:02.991782 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:21:02.991842 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:21:02.992281 26409 log.cc:824] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:02.994186 26409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f: No bootstrap required, opened a new log
I20260928 21:21:02.997694 26409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 21:21:02.997946 26409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:02.998011 26409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ae83ce99fd4dd0b2c9e7d74491894f, State: Initialized, Role: FOLLOWER
I20260928 21:21:02.998420 26409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 21:21:02.998543 26409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:02.998606 26409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:02.998704 26409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:02.999607 26409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 21:21:02.999941 26409 leader_election.cc:302] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ae83ce99fd4dd0b2c9e7d74491894f; no voters:
I20260928 21:21:03.000231 26409 leader_election.cc:288] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:03.000341 26419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:03.000538 26419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [term 1 LEADER]: Becoming Leader. State: Replica: 04ae83ce99fd4dd0b2c9e7d74491894f, State: Running, Role: LEADER
I20260928 21:21:03.000828 26419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 21:21:03.001401 26409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:03.002364 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } } }
I20260928 21:21:03.002486 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:03.002740 26421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ae83ce99fd4dd0b2c9e7d74491894f. Latest consensus state: current_term: 1 leader_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ae83ce99fd4dd0b2c9e7d74491894f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } } }
I20260928 21:21:03.002822 26421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:03.002941 26428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:03.006129 26428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:03.012109 26428 catalog_manager.cc:1347] Generated new cluster ID: b7c824e01c8a4e6d8217efaf8770105c
I20260928 21:21:03.012176 26428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:03.021337 26428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:03.021682 26428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:03.025633 26428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04ae83ce99fd4dd0b2c9e7d74491894f: Generated new TSK 0
I20260928 21:21:03.026343 26428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:03.034687 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26341
2026-09-28T21:21:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:03 dist-test-slave-rc9f krb5kdc[26440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:03 dist-test-slave-rc9f krb5kdc[26440](info): set up 2 sockets
Sep 28 21:21:03 dist-test-slave-rc9f krb5kdc[26440](info): commencing operation
krb5kdc: starting...
W20260928 21:21:03.985543 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:04.105119 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:04.213454 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:04.213755 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:04.213798 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:04.217796 26453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:04.217864 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:04.217882 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:04.217897 26453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:04.217911 26453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:04.223229 26453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:04.224269 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:04.225245 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:04.233667 26459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:04.233652 26461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:04.233649 26458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:04.233951 26453 server_base.cc:1034] running on GCE node
I20260928 21:21:04.234558 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:04.235757 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:04.236923 26453 hybrid_clock.cc:648] HybridClock initialized: now 1790630464236901 us; error 43 us; skew 500 ppm
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:04.243346 26453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:04.244351 26453 webserver.cc:460] Webserver started at http://127.0.0.1:38087/ using document root <none> and password file <none>
I20260928 21:21:04.244913 26453 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:04.244983 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:04.245200 26453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:04.247339 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "ae3883a82fb84ac38f780612a9e60ebd"
format_stamp: "Formatted at 2026-09-28 21:21:04 on dist-test-slave-rc9f"
server_key: "83d580cde10d7ac069eecec08fee7006"
server_key_iv: "4495d862244b2a331e54aef63f918564"
server_key_version: "encryptionkey@0"
I20260928 21:21:04.247881 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "ae3883a82fb84ac38f780612a9e60ebd"
format_stamp: "Formatted at 2026-09-28 21:21:04 on dist-test-slave-rc9f"
server_key: "83d580cde10d7ac069eecec08fee7006"
server_key_iv: "4495d862244b2a331e54aef63f918564"
server_key_version: "encryptionkey@0"
I20260928 21:21:04.251685 26453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:21:04.254315 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:04.255637 26453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:21:04.255793 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "ae3883a82fb84ac38f780612a9e60ebd"
format_stamp: "Formatted at 2026-09-28 21:21:04 on dist-test-slave-rc9f"
server_key: "83d580cde10d7ac069eecec08fee7006"
server_key_iv: "4495d862244b2a331e54aef63f918564"
server_key_version: "encryptionkey@0"
I20260928 21:21:04.255915 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:04.291491 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:04.294584 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:04.294772 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:04.301539 26453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42171
I20260928 21:21:04.301555 26520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42171 every 8 connection(s)
I20260928 21:21:04.302537 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:04.305301 26521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:04.311080 26521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: Bootstrap starting.
I20260928 21:21:04.311986 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26453
I20260928 21:21:04.312108 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:04.312355 5195 external_mini_cluster.cc:1428] Setting key a9ffaae7cb2750ea43c4e4eaa5c45a2c
I20260928 21:21:04.313931 26521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:04.314677 26521 log.cc:824] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:04.316932 26521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: No bootstrap required, opened a new log
I20260928 21:21:04.321329 26521 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } }
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:21:04.321615 26521 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:04.321677 26521 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3883a82fb84ac38f780612a9e60ebd, State: Initialized, Role: FOLLOWER
I20260928 21:21:04.322114 26521 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } }
I20260928 21:21:04.322242 26521 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:04.322305 26521 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:04.322402 26521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:04.323340 26521 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } }
I20260928 21:21:04.323683 26521 leader_election.cc:302] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae3883a82fb84ac38f780612a9e60ebd; no voters:
I20260928 21:21:04.324002 26521 leader_election.cc:288] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:04.324087 26526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:04.324258 26526 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [term 1 LEADER]: Becoming Leader. State: Replica: ae3883a82fb84ac38f780612a9e60ebd, State: Running, Role: LEADER
I20260928 21:21:04.324560 26526 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } }
I20260928 21:21:04.325357 26521 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:04.327139 26528 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae3883a82fb84ac38f780612a9e60ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } } }
I20260928 21:21:04.327289 26528 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:04.327483 26527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae3883a82fb84ac38f780612a9e60ebd. Latest consensus state: current_term: 1 leader_uuid: "ae3883a82fb84ac38f780612a9e60ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae3883a82fb84ac38f780612a9e60ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42171 } } }
I20260928 21:21:04.327608 26527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:04.328724 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:04.313441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:38597 (local address 127.0.0.1:42171)
0928 21:21:04.313870 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:04.313887 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:04.313925 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:21:04.314655 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:04.314674 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:04.315099 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:04.315501 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:04.316319 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:04.317430 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:04.318662 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:04.319177 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:04.319335 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:04.322217 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:04.322223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:04.322239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:04.322277 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:04.324103 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:04.324748 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:04.324755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:04.324761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:04.324906 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:04.325219 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:04.325222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:04.325224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:04.325772 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:04.326006 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:04.326632 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 21:21:04.326869 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":141,"threads_started":1}
W20260928 21:21:04.330567 26540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:04.330646 26540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:04.331827 26541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:04.331944 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:04.332576 26541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:04.342293 26541 catalog_manager.cc:1347] Generated new cluster ID: 55db6510ecfb4808b8d165386c5bd0f8
I20260928 21:21:04.342383 26541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:04.350080 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:04.334233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:33469 (local address 127.0.0.1:42171)
0928 21:21:04.334353 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:04.334357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:04.337954 (+ 3597us) server_negotiation.cc:407] Connection header received
0928 21:21:04.338112 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:04.338115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:04.338173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:04.338255 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:04.339087 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:04.339916 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:04.341071 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:04.341430 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:04.341540 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:04.346635 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:04.346639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:04.346641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:04.346674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:04.348058 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:04.348742 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:04.348745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:04.348746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:04.348835 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:04.349166 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:04.349169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:04.349170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:04.349364 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:04.349522 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:04.349813 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:21:04.349913 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:21:04.356700 26541 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:04.357790 26541 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:04.366158 26541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae3883a82fb84ac38f780612a9e60ebd: Generated new TSK 0
I20260928 21:21:04.366945 26541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:04.370951 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26453
W20260928 21:21:04.375717 26438 connection.cc:535] client connection to 127.0.0.1:42171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42171): Connection reset by peer (error 104)
2026-09-28T21:21:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26552](info): set up 2 sockets
Sep 28 21:21:04 dist-test-slave-rc9f krb5kdc[26552](info): commencing operation
krb5kdc: starting...
W20260928 21:21:05.325603 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:05 dist-test-slave-rc9f krb5kdc[26552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:05.444468 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:05.553623 26565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:05.553916 26565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:05.553961 26565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:05.558115 26565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:05.558190 26565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:05.558208 26565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:05.558224 26565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:05.558243 26565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:05.563855 26565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:05.565021 26565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:05.566092 26565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:05.574781 26565 server_base.cc:1034] running on GCE node
W20260928 21:21:05.574717 26573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:05.574798 26571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:05.574717 26570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:05.575601 26565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:05.576829 26565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:05.578038 26565 hybrid_clock.cc:648] HybridClock initialized: now 1790630465578021 us; error 33 us; skew 500 ppm
Sep 28 21:21:05 dist-test-slave-rc9f krb5kdc[26552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:05.581190 26565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:05.582253 26565 webserver.cc:460] Webserver started at http://127.0.0.1:32769/ using document root <none> and password file <none>
I20260928 21:21:05.582820 26565 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:05.582890 26565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:05.583096 26565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:05.585203 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "6adc07d8ac9f4315a96e337559a0dcc1"
format_stamp: "Formatted at 2026-09-28 21:21:05 on dist-test-slave-rc9f"
server_key: "6a0ce7d93a7c64f1424efa0c994f25a6"
server_key_iv: "28cf98c8821278236b1993a4d3036955"
server_key_version: "encryptionkey@0"
I20260928 21:21:05.585791 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "6adc07d8ac9f4315a96e337559a0dcc1"
format_stamp: "Formatted at 2026-09-28 21:21:05 on dist-test-slave-rc9f"
server_key: "6a0ce7d93a7c64f1424efa0c994f25a6"
server_key_iv: "28cf98c8821278236b1993a4d3036955"
server_key_version: "encryptionkey@0"
I20260928 21:21:05.589775 26565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:21:05.592530 26580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:05.594022 26565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:21:05.594194 26565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "6adc07d8ac9f4315a96e337559a0dcc1"
format_stamp: "Formatted at 2026-09-28 21:21:05 on dist-test-slave-rc9f"
server_key: "6a0ce7d93a7c64f1424efa0c994f25a6"
server_key_iv: "28cf98c8821278236b1993a4d3036955"
server_key_version: "encryptionkey@0"
I20260928 21:21:05.594336 26565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:05.618614 26565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:05.622130 26565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:05.622351 26565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:05.629614 26565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46753
I20260928 21:21:05.629650 26632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46753 every 8 connection(s)
I20260928 21:21:05.630646 26565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:05.633540 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:05.639380 26633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: Bootstrap starting.
I20260928 21:21:05.640940 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26565
I20260928 21:21:05.641054 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:05.641302 5195 external_mini_cluster.cc:1428] Setting key 4026cdf310564edb6864d026b3650f8c
I20260928 21:21:05.642067 26633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:05.642944 26633 log.cc:824] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:05.644845 26633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: No bootstrap required, opened a new log
I20260928 21:21:05.648335 26633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260928 21:21:05.648571 26633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:05.648604 26633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adc07d8ac9f4315a96e337559a0dcc1, State: Initialized, Role: FOLLOWER
I20260928 21:21:05.649039 26633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260928 21:21:05.649183 26633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:05.649250 26633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:05.649358 26633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:05.650339 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:05.642485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:35553 (local address 127.0.0.1:46753)
0928 21:21:05.642943 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:05.642959 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:05.642996 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:21:05.643843 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:05.643865 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:05.644213 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:05.644537 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:05.644956 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:05.645974 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:05.646860 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:05.647354 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:05.647500 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:05.648509 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:05.648514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:05.648528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:05.648973 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:05.649032 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:05.649056 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:05.649180 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:05.649411 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 21:21:05.649542 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":139,"threads_started":1}
I20260928 21:21:05.650344 26633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260928 21:21:05.650767 26633 leader_election.cc:302] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6adc07d8ac9f4315a96e337559a0dcc1; no voters:
I20260928 21:21:05.651111 26633 leader_election.cc:288] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:05.651201 26638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:05.651443 26638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [term 1 LEADER]: Becoming Leader. State: Replica: 6adc07d8ac9f4315a96e337559a0dcc1, State: Running, Role: LEADER
I20260928 21:21:05.651777 26638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } }
I20260928 21:21:05.652453 26633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:05.652988 26638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adc07d8ac9f4315a96e337559a0dcc1. Latest consensus state: current_term: 1 leader_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260928 21:21:05.653101 26638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:05.653645 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:05.654039 26639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adc07d8ac9f4315a96e337559a0dcc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46753 } } }
I20260928 21:21:05.654189 26639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1 [sys.catalog]: This master's current role is: LEADER
W20260928 21:21:05.657541 26653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:05.657652 26653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:05.657708 26643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:05.658557 26643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:05.660478 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:05.655829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:44245 (local address 127.0.0.1:46753)
0928 21:21:05.655930 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:05.655934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:05.656094 (+ 160us) server_negotiation.cc:407] Connection header received
0928 21:21:05.656219 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:05.656222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:05.656274 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:05.656358 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:05.656636 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:05.657462 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:05.658580 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:05.658945 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:05.659051 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:05.659981 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:05.659985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:05.659988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:05.660003 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:05.660034 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:05.660047 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:05.660136 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:05.660286 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:21:05.660334 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 21:21:05.664472 26643 catalog_manager.cc:1347] Generated new cluster ID: 67783b9af7954e8689836f67af21867f
I20260928 21:21:05.664556 26643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:05.677667 26643 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:05.678833 26643 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:05.692428 26643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6adc07d8ac9f4315a96e337559a0dcc1: Generated new TSK 0
I20260928 21:21:05.693193 26643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:05.698771 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26565
W20260928 21:21:05.703303 26550 connection.cc:535] client connection to 127.0.0.1:46753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46753): Connection reset by peer (error 104)
2026-09-28T21:21:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:05 dist-test-slave-rc9f krb5kdc[26664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:05 dist-test-slave-rc9f krb5kdc[26664](info): set up 2 sockets
Sep 28 21:21:05 dist-test-slave-rc9f krb5kdc[26664](info): commencing operation
krb5kdc: starting...
W20260928 21:21:06.653506 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:06 dist-test-slave-rc9f krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:21:06.774897 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:06.886065 26677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:06.886377 26677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:06.886438 26677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:06.890626 26677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:06.890712 26677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:06.890739 26677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:06.890761 26677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:06.890787 26677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:06.896410 26677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:06.897554 26677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:06.898664 26677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:06.907188 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:06.907174 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:06.907188 26685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:06.907366 26677 server_base.cc:1034] running on GCE node
I20260928 21:21:06.908130 26677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:06.909325 26677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:06.910511 26677 hybrid_clock.cc:648] HybridClock initialized: now 1790630466910491 us; error 36 us; skew 500 ppm
Sep 28 21:21:06 dist-test-slave-rc9f krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:06.913264 26677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:21:06.914239 26677 webserver.cc:460] Webserver started at http://127.0.0.1:33135/ using document root <none> and password file <none>
I20260928 21:21:06.914798 26677 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:06.914863 26677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:06.915062 26677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:06.917171 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "0519017ea8b34449b7b98efa2f2875b5"
format_stamp: "Formatted at 2026-09-28 21:21:06 on dist-test-slave-rc9f"
server_key: "677a836998422cb873161dac962561ad"
server_key_iv: "f6e98da174d48463d01576e50295159f"
server_key_version: "encryptionkey@0"
I20260928 21:21:06.917773 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "0519017ea8b34449b7b98efa2f2875b5"
format_stamp: "Formatted at 2026-09-28 21:21:06 on dist-test-slave-rc9f"
server_key: "677a836998422cb873161dac962561ad"
server_key_iv: "f6e98da174d48463d01576e50295159f"
server_key_version: "encryptionkey@0"
I20260928 21:21:06.921479 26677 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:21:06.923916 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:06.925253 26677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:21:06.925391 26677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "0519017ea8b34449b7b98efa2f2875b5"
format_stamp: "Formatted at 2026-09-28 21:21:06 on dist-test-slave-rc9f"
server_key: "677a836998422cb873161dac962561ad"
server_key_iv: "f6e98da174d48463d01576e50295159f"
server_key_version: "encryptionkey@0"
I20260928 21:21:06.925501 26677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:06.930258 26677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:06.933435 26677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:06.933672 26677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:06.940747 26677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33035
I20260928 21:21:06.940770 26744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33035 every 8 connection(s)
I20260928 21:21:06.942010 26677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:06.945850 26745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:06.951311 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26677
I20260928 21:21:06.951467 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:06.951748 5195 external_mini_cluster.cc:1428] Setting key 4d50a943b2680692593c3786bc0f4b87
I20260928 21:21:06.953361 26745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: Bootstrap starting.
I20260928 21:21:06.956439 26745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:06.957374 26745 log.cc:824] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:06.958204 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:06.953324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:42891 (local address 127.0.0.1:33035)
0928 21:21:06.954100 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:06.954144 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:06.954215 (+ 71us) server_negotiation.cc:407] Connection header received
0928 21:21:06.955191 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:06.955217 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:06.955610 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:06.956014 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:06.956720 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:06.956726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:06.956745 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:06.956757 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:06.956824 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:06.956849 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:06.956965 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:06.957100 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:21:06.957271 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":191,"threads_started":1}
I20260928 21:21:06.959653 26745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: No bootstrap required, opened a new log
I20260928 21:21:06.961310 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:06.963842 26745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 21:21:06.964175 26745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:06.964237 26745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0519017ea8b34449b7b98efa2f2875b5, State: Initialized, Role: FOLLOWER
I20260928 21:21:06.964741 26745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 21:21:06.964918 26745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:06.965003 26745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:06.965111 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:06.963420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.50:41223 (local address 127.0.0.1:33035)
0928 21:21:06.963544 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:06.963549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:06.963683 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:21:06.963801 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:06.963803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:06.963836 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:06.963903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:06.964656 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:06.964659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:06.964660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:06.964671 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:06.964698 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:06.964708 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:06.964778 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:06.964863 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 21:21:06.964906 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:21:06.965126 26745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:06.966432 26745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 21:21:06.966780 26745 leader_election.cc:302] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0519017ea8b34449b7b98efa2f2875b5; no voters:
I20260928 21:21:06.967097 26745 leader_election.cc:288] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:06.967183 26755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:06.967373 26755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0519017ea8b34449b7b98efa2f2875b5, State: Running, Role: LEADER
I20260928 21:21:06.967705 26755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260928 21:21:06.968325 26745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:06.969359 26757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0519017ea8b34449b7b98efa2f2875b5. Latest consensus state: current_term: 1 leader_uuid: "0519017ea8b34449b7b98efa2f2875b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260928 21:21:06.969352 26756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0519017ea8b34449b7b98efa2f2875b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0519017ea8b34449b7b98efa2f2875b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260928 21:21:06.969503 26757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:06.969511 26756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5 [sys.catalog]: This master's current role is: LEADER
W20260928 21:21:06.973179 26770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:06.973264 26770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:06.973338 26764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:06.974112 26764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:06.979816 26764 catalog_manager.cc:1347] Generated new cluster ID: 71ce4f80ed8549a6846fcb439de3c3a4
I20260928 21:21:06.979916 26764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:06.998775 26764 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:06.999146 26764 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:07.005214 26764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0519017ea8b34449b7b98efa2f2875b5: Generated new TSK 0
I20260928 21:21:07.006006 26764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:07.010450 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26677
2026-09-28T21:21:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:07 dist-test-slave-rc9f krb5kdc[26776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:07 dist-test-slave-rc9f krb5kdc[26776](info): set up 2 sockets
Sep 28 21:21:07 dist-test-slave-rc9f krb5kdc[26776](info): commencing operation
krb5kdc: starting...
W20260928 21:21:07.965610 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:08.088172 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42749
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:08.199728 26789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:08.200048 26789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:08.200109 26789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:08.204380 26789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:08.204460 26789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:08.204487 26789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:08.204509 26789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:08.204537 26789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:08.210176 26789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42749
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:08.211349 26789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:08.212440 26789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:08.221308 26795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:08.221329 26794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:08.221324 26797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:08.221740 26789 server_base.cc:1034] running on GCE node
I20260928 21:21:08.222332 26789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:08.223493 26789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:08.224661 26789 hybrid_clock.cc:648] HybridClock initialized: now 1790630468224644 us; error 30 us; skew 500 ppm
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:08.227502 26789 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:08.228567 26789 webserver.cc:460] Webserver started at http://127.5.18.254:46383/ using document root <none> and password file <none>
I20260928 21:21:08.229185 26789 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:08.229270 26789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:08.229493 26789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:08.231760 26789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "244ce92217774bb8ae5da2ad10ec8313"
format_stamp: "Formatted at 2026-09-28 21:21:08 on dist-test-slave-rc9f"
server_key: "796be8164a0a89c5f529731ee7f1bd44"
server_key_iv: "f05ae62e459b6c82951c52ed22dd05ad"
server_key_version: "encryptionkey@0"
I20260928 21:21:08.232347 26789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "244ce92217774bb8ae5da2ad10ec8313"
format_stamp: "Formatted at 2026-09-28 21:21:08 on dist-test-slave-rc9f"
server_key: "796be8164a0a89c5f529731ee7f1bd44"
server_key_iv: "f05ae62e459b6c82951c52ed22dd05ad"
server_key_version: "encryptionkey@0"
I20260928 21:21:08.236325 26789 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:21:08.238984 26804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:08.240309 26789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:21:08.240442 26789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "244ce92217774bb8ae5da2ad10ec8313"
format_stamp: "Formatted at 2026-09-28 21:21:08 on dist-test-slave-rc9f"
server_key: "796be8164a0a89c5f529731ee7f1bd44"
server_key_iv: "f05ae62e459b6c82951c52ed22dd05ad"
server_key_version: "encryptionkey@0"
I20260928 21:21:08.240554 26789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:08.271749 26789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:08.274811 26789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:08.275002 26789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:08.282168 26789 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:42749
I20260928 21:21:08.282192 26856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:42749 every 8 connection(s)
I20260928 21:21:08.283159 26789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:08.284981 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26789
I20260928 21:21:08.285111 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:08.285382 5195 external_mini_cluster.cc:1428] Setting key 5341c23c6020a3efdf035934cddb976e
I20260928 21:21:08.286218 26857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:08.293052 26857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: Bootstrap starting.
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:08.295969 26857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:08.296835 26857 log.cc:824] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:08.299083 26857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: No bootstrap required, opened a new log
I20260928 21:21:08.300680 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:08.286833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51241 (local address 127.5.18.254:42749)
0928 21:21:08.287301 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:08.287320 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:08.287364 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:21:08.288184 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:08.288207 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:08.288655 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:08.289031 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:08.289717 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:08.291188 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:08.292100 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:08.292778 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:08.292937 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:08.295510 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:08.295516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:08.295533 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:08.295567 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:08.297426 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:08.297946 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:08.297952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:08.297957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:08.298088 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:08.298359 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:08.298362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:08.298364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:08.298861 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:08.299065 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:08.299367 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 21:21:08.299723 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":124,"threads_started":1}
I20260928 21:21:08.303547 26857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } }
I20260928 21:21:08.303872 26857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:08.303939 26857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244ce92217774bb8ae5da2ad10ec8313, State: Initialized, Role: FOLLOWER
I20260928 21:21:08.304071 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:08.304411 26857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } }
I20260928 21:21:08.304571 26857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:08.304633 26857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:08.304734 26857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:08.306015 26857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } }
I20260928 21:21:08.306485 26857 leader_election.cc:302] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 244ce92217774bb8ae5da2ad10ec8313; no voters:
I20260928 21:21:08.306862 26857 leader_election.cc:288] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:08.306963 26866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:08.307286 26866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [term 1 LEADER]: Becoming Leader. State: Replica: 244ce92217774bb8ae5da2ad10ec8313, State: Running, Role: LEADER
I20260928 21:21:08.307706 26866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } }
I20260928 21:21:08.308377 26857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:08.309118 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "244ce92217774bb8ae5da2ad10ec8313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } } }
I20260928 21:21:08.309303 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:08.309551 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 244ce92217774bb8ae5da2ad10ec8313. Latest consensus state: current_term: 1 leader_uuid: "244ce92217774bb8ae5da2ad10ec8313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "244ce92217774bb8ae5da2ad10ec8313" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42749 } } }
I20260928 21:21:08.309710 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313 [sys.catalog]: This master's current role is: LEADER
W20260928 21:21:08.313997 26882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:08.314145 26882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:08.314581 26876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:08.315853 26876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:08.317857 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:08.307584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57911 (local address 127.5.18.254:42749)
0928 21:21:08.307697 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:08.307702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:08.307942 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:21:08.308127 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:08.308130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:08.308178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:08.308255 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:08.309217 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:08.310057 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:08.311358 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:08.311661 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:08.311735 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:08.314000 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:08.314004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:08.314007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:08.314033 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:08.315662 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:08.316238 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:08.316242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:08.316244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:08.316341 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:08.316729 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:08.316733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:08.316735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:08.316963 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:08.317083 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:08.317592 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 21:21:08.317697 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:21:08.322813 26876 catalog_manager.cc:1347] Generated new cluster ID: 8b4072a148534d80b95aeab1dbeabbb9
I20260928 21:21:08.322892 26876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:08.343966 26876 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:08.345602 26876 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:08.365928 26876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 244ce92217774bb8ae5da2ad10ec8313: Generated new TSK 0
I20260928 21:21:08.366870 26876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:08.374816 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26789
W20260928 21:21:08.380162 26774 connection.cc:535] client connection to 127.5.18.254:42749 recv error: Network error: failed to read from TLS socket (remote: 127.5.18.254:42749): Connection reset by peer (error 104)
2026-09-28T21:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26888](info): set up 2 sockets
Sep 28 21:21:08 dist-test-slave-rc9f krb5kdc[26888](info): commencing operation
krb5kdc: starting...
W20260928 21:21:09.333540 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:09 dist-test-slave-rc9f krb5kdc[26888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:09.457226 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42341
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:42341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:09.567739 26901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:09.568050 26901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:09.568092 26901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:09.572360 26901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:09.572453 26901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:09.572474 26901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:09.572489 26901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:09.572504 26901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:09.578200 26901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:42341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:09.579392 26901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:09.580482 26901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:09.589295 26909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:09.589334 26901 server_base.cc:1034] running on GCE node
W20260928 21:21:09.589299 26906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:09.589272 26907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:09.590167 26901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:09.591316 26901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:09.592506 26901 hybrid_clock.cc:648] HybridClock initialized: now 1790630469592486 us; error 34 us; skew 500 ppm
Sep 28 21:21:09 dist-test-slave-rc9f krb5kdc[26888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630469, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:09.601842 26901 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:09.602978 26901 webserver.cc:460] Webserver started at http://127.5.18.254:34109/ using document root <none> and password file <none>
I20260928 21:21:09.603565 26901 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:09.603639 26901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:09.603848 26901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:09.606060 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "0d00aba313da468fba72cb210c12f984"
format_stamp: "Formatted at 2026-09-28 21:21:09 on dist-test-slave-rc9f"
server_key: "05172a72004718c88b1189ea47d28dc6"
server_key_iv: "0903b192df4f220aaae3b61a8564b3d8"
server_key_version: "encryptionkey@0"
I20260928 21:21:09.606616 26901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "0d00aba313da468fba72cb210c12f984"
format_stamp: "Formatted at 2026-09-28 21:21:09 on dist-test-slave-rc9f"
server_key: "05172a72004718c88b1189ea47d28dc6"
server_key_iv: "0903b192df4f220aaae3b61a8564b3d8"
server_key_version: "encryptionkey@0"
I20260928 21:21:09.610642 26901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:21:09.613245 26916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:09.614651 26901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:21:09.614777 26901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "0d00aba313da468fba72cb210c12f984"
format_stamp: "Formatted at 2026-09-28 21:21:09 on dist-test-slave-rc9f"
server_key: "05172a72004718c88b1189ea47d28dc6"
server_key_iv: "0903b192df4f220aaae3b61a8564b3d8"
server_key_version: "encryptionkey@0"
I20260928 21:21:09.614892 26901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:09.630153 26901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:09.633798 26901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:09.633972 26901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:09.641515 26901 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:42341
I20260928 21:21:09.641721 26968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:42341 every 8 connection(s)
I20260928 21:21:09.642668 26901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:09.643654 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 26901
I20260928 21:21:09.643823 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:09.644146 5195 external_mini_cluster.cc:1428] Setting key 2f3d00582a6d32e2a13ba3c06df8a7ec
I20260928 21:21:09.646382 26969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:09.653662 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984: Bootstrap starting.
I20260928 21:21:09.654872 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:09.645557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39273 (local address 127.5.18.254:42341)
0928 21:21:09.646047 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:09.646069 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:09.646113 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:21:09.646986 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:09.647022 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:09.647491 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:09.647904 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:09.648332 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:09.649904 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:09.650850 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:09.651540 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:09.651734 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:09.652674 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:09.652681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:09.652708 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:09.653216 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:09.653294 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:09.653325 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:09.653461 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:09.653725 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:21:09.653881 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260928 21:21:09.656365 26969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:09.657186 26969 log.cc:824] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:09.658102 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:09.659916 26969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984: No bootstrap required, opened a new log
I20260928 21:21:09.664564 26969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } }
I20260928 21:21:09.664927 26969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:09.665035 26969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d00aba313da468fba72cb210c12f984, State: Initialized, Role: FOLLOWER
I20260928 21:21:09.665185 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:09.660682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57323 (local address 127.5.18.254:42341)
0928 21:21:09.660814 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:09.660818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:09.660980 (+ 162us) server_negotiation.cc:407] Connection header received
0928 21:21:09.661082 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:09.661085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:09.661133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:09.661217 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:09.661474 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:09.662307 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:09.663197 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:09.663567 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:09.663678 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:09.664605 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:09.664609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:09.664612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:09.664629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:09.664660 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:09.664673 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:09.664776 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:09.664976 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 21:21:09.665021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:21:09.665557 26969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } }
I20260928 21:21:09.665756 26969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:09.665824 26969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:09.665930 26969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:09.667023 26969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } }
I20260928 21:21:09.667351 26969 leader_election.cc:302] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d00aba313da468fba72cb210c12f984; no voters:
I20260928 21:21:09.667940 26969 leader_election.cc:288] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:09.668078 26979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:09.668292 26979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [term 1 LEADER]: Becoming Leader. State: Replica: 0d00aba313da468fba72cb210c12f984, State: Running, Role: LEADER
I20260928 21:21:09.668649 26979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } }
I20260928 21:21:09.669257 26969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:09.670444 26981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d00aba313da468fba72cb210c12f984. Latest consensus state: current_term: 1 leader_uuid: "0d00aba313da468fba72cb210c12f984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } } }
I20260928 21:21:09.670596 26981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:09.670444 26980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d00aba313da468fba72cb210c12f984" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d00aba313da468fba72cb210c12f984" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 42341 } } }
I20260928 21:21:09.670850 26980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:09.670974 26988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:09.674221 26988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:09.681030 26988 catalog_manager.cc:1347] Generated new cluster ID: 0a00f69865a74ac3864080b89a52d736
I20260928 21:21:09.681133 26988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:09.707752 26988 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:09.708777 26988 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:09.719514 26988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d00aba313da468fba72cb210c12f984: Generated new TSK 0
I20260928 21:21:09.720314 26988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:09.726256 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 26901
W20260928 21:21:09.731338 26886 connection.cc:535] client connection to 127.5.18.254:42341 recv error: Network error: failed to read from TLS socket (remote: 127.5.18.254:42341): Connection reset by peer (error 104)
2026-09-28T21:21:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:09 dist-test-slave-rc9f krb5kdc[27000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:09 dist-test-slave-rc9f krb5kdc[27000](info): set up 2 sockets
Sep 28 21:21:09 dist-test-slave-rc9f krb5kdc[27000](info): commencing operation
krb5kdc: starting...
W20260928 21:21:10.685527 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:10 dist-test-slave-rc9f krb5kdc[27000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:10.800658 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44001
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:44001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:10.909818 27013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:10.910140 27013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:10.910198 27013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:10.914395 27013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:10.914492 27013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:10.914518 27013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:10.914541 27013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:10.914572 27013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:10.920053 27013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:44001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:44001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:10.921170 27013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:10.922227 27013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:10.930475 27019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:10.930558 27018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:10.930540 27013 server_base.cc:1034] running on GCE node
W20260928 21:21:10.930845 27021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:10.931355 27013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:10.932423 27013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:10.933615 27013 hybrid_clock.cc:648] HybridClock initialized: now 1790630470933597 us; error 26 us; skew 500 ppm
Sep 28 21:21:10 dist-test-slave-rc9f krb5kdc[27000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630470, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:10.936381 27013 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:10.937265 27013 webserver.cc:460] Webserver started at http://127.5.18.254:38015/ using document root <none> and password file <none>
I20260928 21:21:10.937855 27013 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:10.937918 27013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:10.938114 27013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:10.940146 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "347ad63db50b42d2a7323b2cf07ed98d"
format_stamp: "Formatted at 2026-09-28 21:21:10 on dist-test-slave-rc9f"
server_key: "f6cc04de38933569f595267f1a158dcf"
server_key_iv: "9982f5142bad02c76123fabbeaf8bb51"
server_key_version: "encryptionkey@0"
I20260928 21:21:10.940680 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "347ad63db50b42d2a7323b2cf07ed98d"
format_stamp: "Formatted at 2026-09-28 21:21:10 on dist-test-slave-rc9f"
server_key: "f6cc04de38933569f595267f1a158dcf"
server_key_iv: "9982f5142bad02c76123fabbeaf8bb51"
server_key_version: "encryptionkey@0"
I20260928 21:21:10.944327 27013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:21:10.946977 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:10.948426 27013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:21:10.948544 27013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "347ad63db50b42d2a7323b2cf07ed98d"
format_stamp: "Formatted at 2026-09-28 21:21:10 on dist-test-slave-rc9f"
server_key: "f6cc04de38933569f595267f1a158dcf"
server_key_iv: "9982f5142bad02c76123fabbeaf8bb51"
server_key_version: "encryptionkey@0"
I20260928 21:21:10.948653 27013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:10.953082 27013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:10.956029 27013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:10.956197 27013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:10.962718 27013 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:44001
I20260928 21:21:10.962785 27080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:44001 every 8 connection(s)
I20260928 21:21:10.963685 27013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:10.966531 27081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:10.966598 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27013
I20260928 21:21:10.966710 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:10.966985 5195 external_mini_cluster.cc:1428] Setting key dce62ef412b91f43dfbf0c55303fa7e5
I20260928 21:21:10.972625 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:10.968269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46073 (local address 127.5.18.254:44001)
0928 21:21:10.968706 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:10.968723 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:10.968766 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:21:10.969839 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:10.969864 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:10.970248 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:10.970626 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:10.971265 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:10.971271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:10.971285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:10.971296 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:10.971353 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:10.971378 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:10.971507 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:10.971572 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:21:10.971721 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260928 21:21:10.973614 27081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d: Bootstrap starting.
I20260928 21:21:10.975471 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:10.976446 27081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:10.977334 27081 log.cc:824] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:10.979221 27084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:10.977664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33319 (local address 127.5.18.254:44001)
0928 21:21:10.977784 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:10.977789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:10.977988 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:21:10.978069 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:10.978072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:10.978109 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:10.978179 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:10.978882 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:10.978885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:10.978887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:10.978897 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:10.978920 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:10.978930 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:10.978992 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:10.979084 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 21:21:10.979113 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:21:10.979548 27081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d: No bootstrap required, opened a new log
I20260928 21:21:10.983076 27081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } }
I20260928 21:21:10.983325 27081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:10.983392 27081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347ad63db50b42d2a7323b2cf07ed98d, State: Initialized, Role: FOLLOWER
I20260928 21:21:10.983786 27081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } }
I20260928 21:21:10.983916 27081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:10.983978 27081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:10.984073 27081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:10.985108 27081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } }
I20260928 21:21:10.985435 27081 leader_election.cc:302] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347ad63db50b42d2a7323b2cf07ed98d; no voters:
I20260928 21:21:10.985750 27081 leader_election.cc:288] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:10.985841 27091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:10.986042 27091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [term 1 LEADER]: Becoming Leader. State: Replica: 347ad63db50b42d2a7323b2cf07ed98d, State: Running, Role: LEADER
I20260928 21:21:10.986342 27091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } }
I20260928 21:21:10.986886 27081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:10.987848 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ad63db50b42d2a7323b2cf07ed98d. Latest consensus state: current_term: 1 leader_uuid: "347ad63db50b42d2a7323b2cf07ed98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } } }
I20260928 21:21:10.987972 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:10.988229 27092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347ad63db50b42d2a7323b2cf07ed98d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ad63db50b42d2a7323b2cf07ed98d" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 44001 } } }
I20260928 21:21:10.988320 27092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:10.988417 27100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:10.991214 27100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:10.996640 27100 catalog_manager.cc:1347] Generated new cluster ID: da83b1933a5d4f838ba0f21608298f2c
I20260928 21:21:10.996721 27100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:11.012056 27100 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:11.012435 27100 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:11.018862 27100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347ad63db50b42d2a7323b2cf07ed98d: Generated new TSK 0
I20260928 21:21:11.019730 27100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:11.024230 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27013
2026-09-28T21:21:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:11 dist-test-slave-rc9f krb5kdc[27112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:11 dist-test-slave-rc9f krb5kdc[27112](info): set up 2 sockets
Sep 28 21:21:11 dist-test-slave-rc9f krb5kdc[27112](info): commencing operation
krb5kdc: starting...
W20260928 21:21:11.965453 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:12.082515 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37257
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:12.190783 27125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:12.191048 27125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:12.191088 27125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:12.195106 27125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:12.195174 27125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:12.195192 27125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:12.195209 27125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:12.195221 27125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:12.200466 27125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:37257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:12.201529 27125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:12.202548 27125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:12.211025 27130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:12.211025 27133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:12.211153 27125 server_base.cc:1034] running on GCE node
W20260928 21:21:12.211025 27131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:12.211987 27125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:12.213122 27125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:12.214277 27125 hybrid_clock.cc:648] HybridClock initialized: now 1790630472214251 us; error 41 us; skew 500 ppm
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630472, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:12.216958 27125 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:12.217900 27125 webserver.cc:460] Webserver started at http://127.5.18.254:33625/ using document root <none> and password file <none>
I20260928 21:21:12.218420 27125 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:12.218483 27125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:12.218678 27125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:12.220664 27125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "7ee536a667b644c284dc7cfab31879e3"
format_stamp: "Formatted at 2026-09-28 21:21:12 on dist-test-slave-rc9f"
server_key: "fc8311baa9a18602503d07feb81489fc"
server_key_iv: "0fb1ddc7852ce5354279e6c839f85584"
server_key_version: "encryptionkey@0"
I20260928 21:21:12.221196 27125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "7ee536a667b644c284dc7cfab31879e3"
format_stamp: "Formatted at 2026-09-28 21:21:12 on dist-test-slave-rc9f"
server_key: "fc8311baa9a18602503d07feb81489fc"
server_key_iv: "0fb1ddc7852ce5354279e6c839f85584"
server_key_version: "encryptionkey@0"
I20260928 21:21:12.224777 27125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:21:12.227262 27140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:12.228516 27125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:12.228643 27125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "7ee536a667b644c284dc7cfab31879e3"
format_stamp: "Formatted at 2026-09-28 21:21:12 on dist-test-slave-rc9f"
server_key: "fc8311baa9a18602503d07feb81489fc"
server_key_iv: "0fb1ddc7852ce5354279e6c839f85584"
server_key_version: "encryptionkey@0"
I20260928 21:21:12.228744 27125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:12.261008 27125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:12.264391 27125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:12.264585 27125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:12.271414 27125 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:37257
I20260928 21:21:12.271435 27192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:37257 every 8 connection(s)
I20260928 21:21:12.272360 27125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:12.275102 27193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:12.278760 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27125
I20260928 21:21:12.278875 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:12.279124 5195 external_mini_cluster.cc:1428] Setting key d6a93b90838bac287a172dd4923ea3d6
I20260928 21:21:12.280848 27193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3: Bootstrap starting.
I20260928 21:21:12.283283 27193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:12.283988 27193 log.cc:824] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:12.286067 27193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3: No bootstrap required, opened a new log
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:12.290320 27193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } }
I20260928 21:21:12.290596 27193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:12.290663 27193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ee536a667b644c284dc7cfab31879e3, State: Initialized, Role: FOLLOWER
I20260928 21:21:12.291137 27193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } }
I20260928 21:21:12.291292 27193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:12.291366 27193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:12.291471 27193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:12.292505 27193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } }
I20260928 21:21:12.292851 27193 leader_election.cc:302] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ee536a667b644c284dc7cfab31879e3; no voters:
I20260928 21:21:12.293154 27193 leader_election.cc:288] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:12.293274 27198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:12.293494 27198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7ee536a667b644c284dc7cfab31879e3, State: Running, Role: LEADER
I20260928 21:21:12.293713 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:12.280511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54097 (local address 127.5.18.254:37257)
0928 21:21:12.280898 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:12.280914 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:12.280948 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:21:12.281598 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:12.281621 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:12.281974 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:12.282251 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:12.283144 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:12.284185 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:12.285216 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:12.285739 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:12.285865 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:12.288601 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:12.288606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:12.288621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:12.288649 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:12.290400 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:12.290940 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:12.290947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:12.290953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:12.291110 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:12.291439 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:12.291443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:12.291444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:12.291936 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:12.292103 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:12.292396 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 21:21:12.292685 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
I20260928 21:21:12.294404 27193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:12.293840 27198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } }
I20260928 21:21:12.296288 27199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ee536a667b644c284dc7cfab31879e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } } }
I20260928 21:21:12.296417 27199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:12.296496 27200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ee536a667b644c284dc7cfab31879e3. Latest consensus state: current_term: 1 leader_uuid: "7ee536a667b644c284dc7cfab31879e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ee536a667b644c284dc7cfab31879e3" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 37257 } } }
I20260928 21:21:12.296816 27200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:12.296895 27213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:12.296890 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:12.299777 27213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:12.305006 27213 catalog_manager.cc:1347] Generated new cluster ID: c084c670f7c44fd0973f975cf4606788
I20260928 21:21:12.305068 27213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:12.308506 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:12.299199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39749 (local address 127.5.18.254:37257)
0928 21:21:12.299318 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:12.299323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:12.299569 (+ 246us) server_negotiation.cc:407] Connection header received
0928 21:21:12.299786 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:12.299790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:12.299841 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:12.299894 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:12.300924 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:12.301736 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:12.302799 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:12.303144 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:12.303257 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:12.305096 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:12.305100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:12.305103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:12.305130 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:12.306835 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:12.307280 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:12.307284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:12.307286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:12.307381 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:12.307626 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:12.307630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:12.307632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:12.307864 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:12.308000 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:12.308193 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:21:12.308318 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:21:12.325089 27213 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:12.326263 27213 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:12.333743 27213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ee536a667b644c284dc7cfab31879e3: Generated new TSK 0
I20260928 21:21:12.334467 27213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:12.342185 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27125
2026-09-28T21:21:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27224](info): set up 2 sockets
Sep 28 21:21:12 dist-test-slave-rc9f krb5kdc[27224](info): commencing operation
krb5kdc: starting...
W20260928 21:21:13.297505 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:13 dist-test-slave-rc9f krb5kdc[27224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:13.416603 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38623
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:13.525316 27237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:13.525692 27237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:13.525746 27237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:13.529879 27237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:13.529953 27237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:13.529973 27237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:13.529987 27237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:13.530001 27237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:13.535422 27237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:13.536490 27237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:13.537513 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:13.545811 27243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:13.545809 27242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:13.545861 27237 server_base.cc:1034] running on GCE node
W20260928 21:21:13.545874 27245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:13.546626 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:13.547693 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:13.548846 27237 hybrid_clock.cc:648] HybridClock initialized: now 1790630473548828 us; error 29 us; skew 500 ppm
Sep 28 21:21:13 dist-test-slave-rc9f krb5kdc[27224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630473, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:13.551404 27237 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:13.552248 27237 webserver.cc:460] Webserver started at http://127.5.18.254:44655/ using document root <none> and password file <none>
I20260928 21:21:13.552769 27237 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:13.552831 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:13.553018 27237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:13.555115 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "dd7c57947eac4980b66acb37f565d451"
format_stamp: "Formatted at 2026-09-28 21:21:13 on dist-test-slave-rc9f"
server_key: "153340abc1d79172b4ae949826e68dc5"
server_key_iv: "314426099e3321cad52bf071759104ce"
server_key_version: "encryptionkey@0"
I20260928 21:21:13.555675 27237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "dd7c57947eac4980b66acb37f565d451"
format_stamp: "Formatted at 2026-09-28 21:21:13 on dist-test-slave-rc9f"
server_key: "153340abc1d79172b4ae949826e68dc5"
server_key_iv: "314426099e3321cad52bf071759104ce"
server_key_version: "encryptionkey@0"
I20260928 21:21:13.559429 27237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:21:13.561844 27252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:13.563030 27237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:21:13.563144 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "dd7c57947eac4980b66acb37f565d451"
format_stamp: "Formatted at 2026-09-28 21:21:13 on dist-test-slave-rc9f"
server_key: "153340abc1d79172b4ae949826e68dc5"
server_key_iv: "314426099e3321cad52bf071759104ce"
server_key_version: "encryptionkey@0"
I20260928 21:21:13.563244 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:13.588934 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:13.591709 27237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:13.591892 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:13.598548 27237 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38623
I20260928 21:21:13.598563 27304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38623 every 8 connection(s)
I20260928 21:21:13.599490 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:13.602295 27305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:13.602356 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27237
I20260928 21:21:13.602475 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:13.602727 5195 external_mini_cluster.cc:1428] Setting key 3f196a81ebfdbb589e84beb20ccca7ef
I20260928 21:21:13.608788 27305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451: Bootstrap starting.
I20260928 21:21:13.611317 27305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:13.611428 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:13.604093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.5.18.254:38623)
0928 21:21:13.604498 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:13.604514 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:13.604548 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:21:13.605218 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:13.605235 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:13.605611 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:13.605902 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:13.606224 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:13.607186 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:13.608361 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:13.608809 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:13.608938 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:13.609803 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:13.609808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:13.609819 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:13.610182 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:13.610235 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:13.610255 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:13.610327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:13.610476 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 21:21:13.610678 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260928 21:21:13.612064 27305 log.cc:824] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:13.614171 27305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451: No bootstrap required, opened a new log
I20260928 21:21:13.614282 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:13.617954 27305 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } }
I20260928 21:21:13.618189 27305 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:13.618247 27305 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7c57947eac4980b66acb37f565d451, State: Initialized, Role: FOLLOWER
I20260928 21:21:13.618651 27305 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } }
I20260928 21:21:13.618778 27305 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:13.618839 27305 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:13.618940 27305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:13.619750 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:13.616130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48505 (local address 127.5.18.254:38623)
0928 21:21:13.616232 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:13.616237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:13.616406 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:21:13.616516 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:13.616520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:13.616561 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:13.616619 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:13.616857 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:13.617374 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:13.618200 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:13.618533 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:13.618623 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:13.619387 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:13.619391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:13.619394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:13.619407 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:13.619432 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:13.619441 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:13.619496 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:13.619578 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:21:13.619626 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:21:13.619946 27305 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } }
I20260928 21:21:13.620292 27305 leader_election.cc:302] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd7c57947eac4980b66acb37f565d451; no voters:
I20260928 21:21:13.620627 27305 leader_election.cc:288] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:13.620716 27315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:13.620927 27315 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [term 1 LEADER]: Becoming Leader. State: Replica: dd7c57947eac4980b66acb37f565d451, State: Running, Role: LEADER
I20260928 21:21:13.621215 27315 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } }
I20260928 21:21:13.621855 27305 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:13.622794 27316 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd7c57947eac4980b66acb37f565d451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } } }
I20260928 21:21:13.622789 27317 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd7c57947eac4980b66acb37f565d451. Latest consensus state: current_term: 1 leader_uuid: "dd7c57947eac4980b66acb37f565d451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd7c57947eac4980b66acb37f565d451" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38623 } } }
I20260928 21:21:13.622922 27316 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:13.622922 27317 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451 [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:13.623379 27324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:13.626137 27324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:13.631637 27324 catalog_manager.cc:1347] Generated new cluster ID: 3ee8af4046a246e29202b23705e91e89
I20260928 21:21:13.631698 27324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:13.644325 27324 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:13.645684 27324 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:13.653039 27324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd7c57947eac4980b66acb37f565d451: Generated new TSK 0
I20260928 21:21:13.653971 27324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:13.666270 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27237
2026-09-28T21:21:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:13 dist-test-slave-rc9f krb5kdc[27336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:13 dist-test-slave-rc9f krb5kdc[27336](info): set up 2 sockets
Sep 28 21:21:13 dist-test-slave-rc9f krb5kdc[27336](info): commencing operation
krb5kdc: starting...
W20260928 21:21:14.613489 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:14 dist-test-slave-rc9f krb5kdc[27336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:14.730584 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:39955
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:14.836473 27349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:14.836731 27349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:14.836772 27349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:14.840749 27349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:14.840826 27349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:14.840850 27349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:14.840866 27349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:14.840880 27349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:14.846179 27349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:39955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:14.847189 27349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:14.848145 27349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:14.856251 27354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:14.856268 27355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:14.856328 27357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:14.856319 27349 server_base.cc:1034] running on GCE node
I20260928 21:21:14.856976 27349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:14.858098 27349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:14.859287 27349 hybrid_clock.cc:648] HybridClock initialized: now 1790630474859267 us; error 32 us; skew 500 ppm
Sep 28 21:21:14 dist-test-slave-rc9f krb5kdc[27336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630474, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:14.867272 27349 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:14.868273 27349 webserver.cc:460] Webserver started at http://127.5.18.254:43831/ using document root <none> and password file <none>
I20260928 21:21:14.868860 27349 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:14.868933 27349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:14.869140 27349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:14.871279 27349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "4cc117782f43442f84bdd9deff8f59bd"
format_stamp: "Formatted at 2026-09-28 21:21:14 on dist-test-slave-rc9f"
server_key: "0e4b1a6abef26b36eaef5228df75748b"
server_key_iv: "ab143de5bbd9bfac90fe53eec7bbe321"
server_key_version: "encryptionkey@0"
I20260928 21:21:14.871805 27349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "4cc117782f43442f84bdd9deff8f59bd"
format_stamp: "Formatted at 2026-09-28 21:21:14 on dist-test-slave-rc9f"
server_key: "0e4b1a6abef26b36eaef5228df75748b"
server_key_iv: "ab143de5bbd9bfac90fe53eec7bbe321"
server_key_version: "encryptionkey@0"
I20260928 21:21:14.875705 27349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 21:21:14.878233 27364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:14.879421 27349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:21:14.879544 27349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "4cc117782f43442f84bdd9deff8f59bd"
format_stamp: "Formatted at 2026-09-28 21:21:14 on dist-test-slave-rc9f"
server_key: "0e4b1a6abef26b36eaef5228df75748b"
server_key_iv: "ab143de5bbd9bfac90fe53eec7bbe321"
server_key_version: "encryptionkey@0"
I20260928 21:21:14.879638 27349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:14.884089 27349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:14.886858 27349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:14.887014 27349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:14.893498 27349 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:39955
I20260928 21:21:14.893534 27416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:39955 every 8 connection(s)
I20260928 21:21:14.894466 27349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:14.896191 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27349
I20260928 21:21:14.896334 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:14.896589 5195 external_mini_cluster.cc:1428] Setting key 2461304094d8411cc0c57802f55f5ea1
I20260928 21:21:14.897315 27417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:14.901489 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:14.897862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47303 (local address 127.5.18.254:39955)
0928 21:21:14.898248 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:14.898263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:14.898299 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:21:14.898931 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:14.898947 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:14.899221 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:14.899498 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:14.900325 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:14.900329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:14.900340 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:14.900350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:14.900400 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:14.900419 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:14.900519 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:14.900626 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:21:14.900767 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260928 21:21:14.903502 27417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: Bootstrap starting.
I20260928 21:21:14.904264 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:14.905814 27417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:14.906595 27417 log.cc:824] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:14.907701 27420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:14.906180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48555 (local address 127.5.18.254:39955)
0928 21:21:14.906295 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:14.906308 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:14.906504 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:21:14.906588 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:14.906590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:14.906630 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:14.906713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:14.907373 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:14.907376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:14.907379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:21:14.907390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:14.907415 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:21:14.907426 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:14.907511 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:14.907554 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 21:21:14.907589 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:21:14.908681 27417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: No bootstrap required, opened a new log
I20260928 21:21:14.912112 27417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } }
I20260928 21:21:14.912333 27417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:21:14.912374 27417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc117782f43442f84bdd9deff8f59bd, State: Initialized, Role: FOLLOWER
I20260928 21:21:14.912760 27417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } }
I20260928 21:21:14.912897 27417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:21:14.912959 27417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:21:14.913056 27417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:14.914052 27417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } }
I20260928 21:21:14.914342 27417 leader_election.cc:302] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cc117782f43442f84bdd9deff8f59bd; no voters:
I20260928 21:21:14.914608 27417 leader_election.cc:288] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:21:14.914712 27427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:14.914958 27427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [term 1 LEADER]: Becoming Leader. State: Replica: 4cc117782f43442f84bdd9deff8f59bd, State: Running, Role: LEADER
I20260928 21:21:14.915262 27427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } }
I20260928 21:21:14.915748 27417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:14.916947 27428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cc117782f43442f84bdd9deff8f59bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } } }
I20260928 21:21:14.916982 27429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cc117782f43442f84bdd9deff8f59bd. Latest consensus state: current_term: 1 leader_uuid: "4cc117782f43442f84bdd9deff8f59bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cc117782f43442f84bdd9deff8f59bd" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 39955 } } }
I20260928 21:21:14.917146 27428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:14.917171 27429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd [sys.catalog]: This master's current role is: LEADER
W20260928 21:21:14.920101 27442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:14.920168 27442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:14.920244 27440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:14.920897 27440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:14.926433 27440 catalog_manager.cc:1347] Generated new cluster ID: 212673a8655f4d90a7d5669897bee02f
I20260928 21:21:14.926503 27440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:14.951164 27440 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:14.951460 27440 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:14.955336 27440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cc117782f43442f84bdd9deff8f59bd: Generated new TSK 0
I20260928 21:21:14.956007 27440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:21:14.961048 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27349
2026-09-28T21:21:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1294 ms)
[----------] 18 tests from AuthTokenIssuingTest (23835 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:15 dist-test-slave-rc9f krb5kdc[27448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:15 dist-test-slave-rc9f krb5kdc[27448](info): set up 2 sockets
Sep 28 21:21:15 dist-test-slave-rc9f krb5kdc[27448](info): commencing operation
krb5kdc: starting...
W20260928 21:21:15.905601 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:15 dist-test-slave-rc9f krb5kdc[27448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:16.022570 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38607
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:16.130477 27461 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:16.130785 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:16.130810 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:16.130846 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:16.134857 27461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:16.134927 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:16.134945 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:16.134960 27461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:16.134975 27461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:16.140301 27461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:38607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:16.141357 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:16.142395 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:16.151027 27461 server_base.cc:1034] running on GCE node
W20260928 21:21:16.150985 27466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.150985 27469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.150990 27467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:16.151804 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:16.152985 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:16.154142 27461 hybrid_clock.cc:648] HybridClock initialized: now 1790630476154124 us; error 34 us; skew 500 ppm
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:16.156965 27461 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:16.157917 27461 webserver.cc:460] Webserver started at http://127.5.18.254:33513/ using document root <none> and password file <none>
I20260928 21:21:16.158455 27461 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:16.158512 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:16.158676 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:16.160722 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "a8de7da0d45a408aaeb3ba726b0546db"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "5163177de72434cc9f17e4778b09be9f"
server_key_iv: "bd4a7a2f123704d587ed0cd0e16d6ac4"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.161219 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "a8de7da0d45a408aaeb3ba726b0546db"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "5163177de72434cc9f17e4778b09be9f"
server_key_iv: "bd4a7a2f123704d587ed0cd0e16d6ac4"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.164971 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:21:16.167399 27476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:16.168723 27461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:21:16.168859 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "a8de7da0d45a408aaeb3ba726b0546db"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "5163177de72434cc9f17e4778b09be9f"
server_key_iv: "bd4a7a2f123704d587ed0cd0e16d6ac4"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.168954 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:16.198287 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:16.201237 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:16.201399 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:16.208019 27461 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:38607
I20260928 21:21:16.208282 27528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:38607 every 8 connection(s)
I20260928 21:21:16.209003 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:16.211679 27529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:16.215418 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.218713 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27461
I20260928 21:21:16.218832 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:16.219089 5195 external_mini_cluster.cc:1428] Setting key 7b493d57cd0e1ee6b53dce5da12394b5
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.253@KRBTEST.COM" created.
I20260928 21:21:16.230026 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.217663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.5.18.254:38607)
0928 21:21:16.218233 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.218246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.218405 (+ 159us) server_negotiation.cc:407] Connection header received
0928 21:21:16.219551 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.219558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.219794 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.219995 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.220801 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.221705 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.222708 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.223284 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.223456 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.225706 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.225713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.225729 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.225756 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.227275 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.227709 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.227714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.227716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.227819 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.228086 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.228088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.228089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.228463 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.228564 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.228924 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.229100 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":466,"thread_start_us":225,"threads_started":1}
I20260928 21:21:16.230031 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.217663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38607 (local address 127.0.0.1:42254)
0928 21:21:16.218244 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.218304 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.219223 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.220013 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.220017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.220363 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.220645 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.220658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.221841 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.221845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.222575 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.222586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.223431 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.223434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.223535 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.224107 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.224124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.225528 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.227375 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.227383 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.227392 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.227631 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.227919 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.227921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.227923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.228022 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.228558 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.228564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.228702 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.228847 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.229184 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":86,"spinlock_wait_cycles":30592,"thread_start_us":217,"threads_started":1}
I20260928 21:21:16.232672 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.233421 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.233121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53920)
0928 21:21:16.233244 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.233342 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 21:21:16.233649 27478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
I20260928 21:21:16.234544 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.234289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53924)
0928 21:21:16.234408 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.234476 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:21:16.234724 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:40807: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
I20260928 21:21:16.244901 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:40807
--webserver_interface=127.5.18.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:21:16.284013 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } attempt: 1
I20260928 21:21:16.284987 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.284617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53928)
0928 21:21:16.284804 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.284884 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 21:21:16.285892 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.285648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53934)
0928 21:21:16.285762 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.285822 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 21:21:16.286031 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:40807: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
W20260928 21:21:16.352519 27536 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:16.352837 27536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:16.352860 27536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:16.352897 27536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:21:16.354219 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } attempt: 2
I20260928 21:21:16.355062 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.354746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53936)
0928 21:21:16.354904 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.354973 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 21:21:16.355743 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.355546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53942)
0928 21:21:16.355635 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.355682 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:21:16.355907 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:40807: Network error: Client connection negotiation failed: client connection to 127.5.18.253:40807: connect: Connection refused (error 111)
W20260928 21:21:16.357371 27536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:16.357424 27536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:16.357442 27536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:16.357458 27536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:16.357472 27536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:16.362941 27536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.18.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:16.363981 27536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:16.364938 27536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:16.373137 27536 server_base.cc:1034] running on GCE node
W20260928 21:21:16.373199 27542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.373188 27541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.373385 27544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:16.373976 27536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:16.375092 27536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:16.376262 27536 hybrid_clock.cc:648] HybridClock initialized: now 1790630476376239 us; error 34 us; skew 500 ppm
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:16.379144 27536 init.cc:377] Logged in from keytab as kudu/127.5.18.253@KRBTEST.COM (short username kudu)
I20260928 21:21:16.380072 27536 webserver.cc:460] Webserver started at http://127.5.18.253:44787/ using document root <none> and password file <none>
I20260928 21:21:16.380614 27536 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:16.380678 27536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:16.380878 27536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:16.382930 27536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data/instance:
uuid: "1ea0a47596624614bb752ef6f266be94"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "70e1161bcf484007c3aa5160ac06b6de"
server_key_iv: "c492b487db3e0eb31c507f8a92f1406e"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.383453 27536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal/instance:
uuid: "1ea0a47596624614bb752ef6f266be94"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "70e1161bcf484007c3aa5160ac06b6de"
server_key_iv: "c492b487db3e0eb31c507f8a92f1406e"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.387125 27536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:21:16.389657 27551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:16.390887 27536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:21:16.391008 27536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "1ea0a47596624614bb752ef6f266be94"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "70e1161bcf484007c3aa5160ac06b6de"
server_key_iv: "c492b487db3e0eb31c507f8a92f1406e"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.391116 27536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:16.401979 27536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:16.404770 27536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:16.404942 27536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:16.411290 27536 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.253:40807
I20260928 21:21:16.411556 27603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.253:40807 every 8 connection(s)
I20260928 21:21:16.412240 27536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
I20260928 21:21:16.414956 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:16.418623 27604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.420902 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27536
I20260928 21:21:16.421032 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-1/wal/instance
I20260928 21:21:16.421325 5195 external_mini_cluster.cc:1428] Setting key 5acb3c31e5626a2de9807b4a862c9cf4
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.18.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.252@KRBTEST.COM" created.
I20260928 21:21:16.432413 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.420408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.5.18.254:38607)
0928 21:21:16.420539 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.420543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.421438 (+ 895us) server_negotiation.cc:407] Connection header received
0928 21:21:16.422635 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.422638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.422687 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.422758 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.423956 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.424733 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.425903 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.426264 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.426389 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.428859 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.428862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.428864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.428895 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.430252 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.430727 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.430731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.430732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.430844 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.431167 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.431169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.431171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.431418 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.431497 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.432112 (+ 615us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.432243 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:21:16.433214 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.420751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38607 (local address 127.0.0.1:42262)
0928 21:21:16.421261 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.421336 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.422408 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.422972 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.422981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.423340 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.423809 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.423822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.424842 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.424844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.425789 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.425798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.426426 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.426429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.426580 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.427172 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.427187 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.428665 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.430375 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.430380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.430392 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.430636 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.430947 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.430949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.430951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.431087 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.431535 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.431541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.431849 (+ 308us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.432031 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.432385 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":121,"threads_started":1}
I20260928 21:21:16.434666 27604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:21:16.445770 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.435152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53950)
0928 21:21:16.435319 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.435332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.435407 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.435997 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.436000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.436130 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.436215 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.436219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.437614 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.437617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.438287 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.438296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.439336 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.439340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.439434 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.439905 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.439927 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.442066 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.443841 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.443845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.443847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.444126 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.444506 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.444508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.444510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.444598 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.445199 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.445202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.445307 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.445417 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.445584 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 21:21:16.445938 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.435252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.5.18.253:40807)
0928 21:21:16.435577 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.435589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.435611 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:21:16.435656 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.435661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.435879 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.436025 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.436311 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.437475 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.438390 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.438933 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.439030 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.442193 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.442204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.442216 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.442237 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.443750 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.444247 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.444254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.444256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.444349 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.444691 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.444694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.444696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.445072 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.445212 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.445525 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.445718 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
I20260928 21:21:16.447041 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } attempt: 3
I20260928 21:21:16.448166 27604 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.449102 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.448583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52416)
0928 21:21:16.448947 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.449025 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
W20260928 21:21:16.449379 27555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
I20260928 21:21:16.450309 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.450107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52426)
0928 21:21:16.450199 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.450245 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 21:21:16.450717 27604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
I20260928 21:21:16.450949 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:33511
--webserver_interface=127.5.18.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:21:16.456784 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.447622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.5.18.253:40807)
0928 21:21:16.448263 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.448269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.448281 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:21:16.448313 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.448316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.448361 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.448437 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.448917 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.449682 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.451298 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.451615 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.451704 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.453945 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.453948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.453950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.453972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.455253 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.455717 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.455720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.455721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.455792 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.456064 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.456066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.456067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.456233 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.456302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.456576 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.456669 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568}
I20260928 21:21:16.456868 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.447555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53966)
0928 21:21:16.447766 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.447782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.447869 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.448488 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.448504 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.448688 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.448801 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.448807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.449809 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.449813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.451157 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.451169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.451715 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.451718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.451806 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.452284 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.452299 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.453808 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.455359 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.455362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.455365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.455617 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.455889 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.455893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.455895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.455986 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.456313 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.456316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.456430 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.456545 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.456695 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 21:21:16.458114 27529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.458880 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.458612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52438)
0928 21:21:16.458742 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.458800 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:21:16.459619 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.459357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52452)
0928 21:21:16.459504 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.459556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 21:21:16.459735 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
I20260928 21:21:16.490900 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 1
I20260928 21:21:16.491796 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.491454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52454)
0928 21:21:16.491627 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.491698 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 21:21:16.492476 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.492263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52464)
0928 21:21:16.492355 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.492410 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:21:16.492599 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
I20260928 21:21:16.499996 27604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 1
I20260928 21:21:16.500876 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.500554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52480)
0928 21:21:16.500729 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.500796 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 21:21:16.501538 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.501334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52484)
0928 21:21:16.501443 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.501485 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:21:16.501704 27604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
W20260928 21:21:16.561069 27611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:16.561359 27611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:16.561408 27611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:16.561472 27611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:16.565461 27611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:16.565537 27611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:16.565585 27611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:16.565609 27611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:16.565642 27611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:16.567751 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 2
I20260928 21:21:16.568656 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.568332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52496)
0928 21:21:16.568494 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.568564 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:21:16.569377 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.569177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52500)
0928 21:21:16.569270 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.569317 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:21:16.569502 27529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
I20260928 21:21:16.569871 27604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 2
I20260928 21:21:16.570480 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.570249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52504)
0928 21:21:16.570369 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.570419 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:21:16.571108 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.570921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52520)
0928 21:21:16.571018 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.571058 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:21:16.571213 27604 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:33511: Network error: Client connection negotiation failed: client connection to 127.5.18.252:33511: connect: Connection refused (error 111)
I20260928 21:21:16.571712 27611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:38607,127.5.18.253:40807,127.5.18.252:33511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.18.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:16.572768 27611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:16.573778 27611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:16.582089 27616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.582090 27619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:16.582093 27617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:16.582461 27611 server_base.cc:1034] running on GCE node
I20260928 21:21:16.583031 27611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:16.584124 27611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:16.585289 27611 hybrid_clock.cc:648] HybridClock initialized: now 1790630476585267 us; error 33 us; skew 500 ppm
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:16.587942 27611 init.cc:377] Logged in from keytab as kudu/127.5.18.252@KRBTEST.COM (short username kudu)
I20260928 21:21:16.588783 27611 webserver.cc:460] Webserver started at http://127.5.18.252:44641/ using document root <none> and password file <none>
I20260928 21:21:16.589305 27611 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:16.589370 27611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:16.589586 27611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:16.591627 27611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data/instance:
uuid: "bcc0ffa635674501a057f32af8c5fa5f"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "b96ff9bb54ce3cd93201ea3c11be2dca"
server_key_iv: "a7eca61ed1ca59b5c4b3f24efa8c6568"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.592177 27611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal/instance:
uuid: "bcc0ffa635674501a057f32af8c5fa5f"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "b96ff9bb54ce3cd93201ea3c11be2dca"
server_key_iv: "a7eca61ed1ca59b5c4b3f24efa8c6568"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.595956 27611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:21:16.598326 27626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:16.599529 27611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:21:16.599645 27611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "bcc0ffa635674501a057f32af8c5fa5f"
format_stamp: "Formatted at 2026-09-28 21:21:16 on dist-test-slave-rc9f"
server_key: "b96ff9bb54ce3cd93201ea3c11be2dca"
server_key_iv: "a7eca61ed1ca59b5c4b3f24efa8c6568"
server_key_version: "encryptionkey@0"
I20260928 21:21:16.599743 27611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:16.632365 27611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:16.635329 27611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:16.635506 27611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:16.642283 27611 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.252:33511
I20260928 21:21:16.642552 27678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.252:33511 every 8 connection(s)
I20260928 21:21:16.643261 27611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
I20260928 21:21:16.646095 27679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:16.647858 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27611
I20260928 21:21:16.648001 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790630206151656-5195-0/minicluster-data/master-2/wal/instance
I20260928 21:21:16.648281 5195 external_mini_cluster.cc:1428] Setting key 9345d3917ee416f3182bc0163b9407e0
I20260928 21:21:16.650403 27679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:16.662138 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.652641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.5.18.254:38607)
0928 21:21:16.652781 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.652785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.653048 (+ 263us) server_negotiation.cc:407] Connection header received
0928 21:21:16.653168 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.653172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.653231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.653311 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.654079 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.654590 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.655470 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.655721 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.655833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.658557 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.658562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.658565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.658596 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.660096 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.660593 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.660596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.660598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.660675 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.661039 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.661043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.661044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.661280 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.661387 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.661826 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.661954 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:21:16.662345 27604 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 3
I20260928 21:21:16.662981 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.652528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:38607 (local address 127.0.0.1:42264)
0928 21:21:16.652922 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.652970 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.653083 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.653398 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.653403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.653806 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.653943 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.653951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.654715 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.654718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.655352 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.655361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.655859 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.655863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.655963 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.656449 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.656471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.658413 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.660225 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.660235 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.660246 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.660509 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.660823 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.660827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.660829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.660957 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.661386 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.661391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.661510 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.661707 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.662043 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":137,"threads_started":1}
I20260928 21:21:16.663460 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.649721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.5.18.252:33511)
0928 21:21:16.650166 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.650184 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.650232 (+ 48us) server_negotiation.cc:407] Connection header received
0928 21:21:16.651221 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.651240 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.651604 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.651920 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.652722 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.653923 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.654791 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.655256 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.655391 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.658029 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.658037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.658050 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.658080 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.660032 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.660528 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.660533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.660534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.660672 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.660981 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.660983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.660985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.661470 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.661682 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.661891 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.663337 (+ 1446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1303,"server-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260928 21:21:16.664263 27679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.668632 27529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } attempt: 3
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:16.672657 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.662768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52536)
0928 21:21:16.662908 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.662923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.663006 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.664446 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.664450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.664637 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.664772 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.664781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.665785 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.665789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.666447 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.666460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.666968 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.666972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.667055 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.667982 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.667993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.669234 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.671134 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.671137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.671140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.671394 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.671680 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.671683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.671685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.671773 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.672167 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.672170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.672267 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.672356 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.672485 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260928 21:21:16.673280 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.662840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52536 (local address 127.5.18.252:33511)
0928 21:21:16.663448 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.663460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.663474 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:21:16.663507 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.663510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.664286 (+ 776us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.664894 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.664992 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.665677 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.666556 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.666845 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.667247 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.669379 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.669383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.669385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.669408 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.671030 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.671508 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.671511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.671513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.671591 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.671874 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.671877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.671879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.672067 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.672930 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.673003 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.673108 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":123,"threads_started":1}
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:21:16.676795 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.664754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.5.18.253:40807)
0928 21:21:16.664844 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.664849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.664964 (+ 115us) server_negotiation.cc:407] Connection header received
0928 21:21:16.665113 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.665115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.665159 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.665201 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.665706 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.666371 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.667480 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.667775 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.667879 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.671999 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.672003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.672006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.672033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.673717 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.675251 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.675254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.675256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.675345 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.675670 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.675674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.675675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.675978 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.676057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.676456 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.676579 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 21:21:16.676803 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: Bootstrap starting.
I20260928 21:21:16.677030 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.665592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.5.18.254:38607)
0928 21:21:16.665696 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.665701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.665837 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:21:16.665949 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.665952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.665997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.666075 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.666742 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.667452 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.668432 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.668743 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.668848 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.672114 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.672118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.672120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.672145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.674531 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.675045 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.675049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.675051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.675364 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.675694 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.675935 (+ 241us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.675936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.676151 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.676374 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.676758 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.676874 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:21:16.677309 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.664681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:40807 (local address 127.0.0.1:53970)
0928 21:21:16.664907 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.664918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.665022 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.665309 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.665311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.665477 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.665611 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.665619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.666487 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.666490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.667333 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.667343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.667894 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.667897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.667981 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.668409 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.668425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.671833 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.674877 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.674880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.674882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.675157 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.675443 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.675447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.675449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.675550 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.676136 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.676140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.676232 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.677011 (+ 779us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.677151 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260928 21:21:16.678915 27679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:16.679951 27604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:16.680727 27532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.669030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52550)
0928 21:21:16.669286 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.669302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.669380 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.669659 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.669662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.669803 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.669907 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.669913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.670922 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.670926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.671694 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.671703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.672315 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.672319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.672399 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.672983 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.672997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.674744 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.679152 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.679155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.679157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.679402 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.679693 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.679697 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.679698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.679782 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.680267 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.680270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.680373 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.680451 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.680576 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 21:21:16.680881 27604 log.cc:824] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:16.681514 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.669263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.5.18.252:33511)
0928 21:21:16.669370 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.669373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.669384 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:21:16.669470 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.669472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.669507 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.670012 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.670093 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.670778 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.671896 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.672203 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.672636 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.674894 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.674898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.674900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.674923 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.678985 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.679518 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.679521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.679523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.679599 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.679892 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.679895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.679897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.680153 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.681218 (+ 1065us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.681292 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.681413 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630476, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:21:16.684393 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: Bootstrap starting.
Sep 28 21:21:16 dist-test-slave-rc9f krb5kdc[27448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:21:16.687217 27529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:16.688014 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.679576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.5.18.252:33511)
0928 21:21:16.679731 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.679737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.679970 (+ 233us) server_negotiation.cc:407] Connection header received
0928 21:21:16.679997 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.680000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.680037 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.680097 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.680557 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.681302 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.682355 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.682667 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.682758 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.684743 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.684746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.684748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.684769 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.686316 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.686756 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.686760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.686762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.686853 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.687147 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.687150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.687152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.687367 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.687468 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.687785 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.687880 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:21:16.688123 27529 log.cc:824] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:16.688014 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.679374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:33511 (local address 127.0.0.1:52560)
0928 21:21:16.679805 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:16.679818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:16.679895 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:16.680187 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:16.680190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:16.680353 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:16.680442 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.680449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.681420 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.681423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.682221 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:16.682231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.682846 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.682849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:16.682916 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.683276 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:16.683290 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:16.684621 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:16.686432 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.686434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.686436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.686646 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.686956 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:16.686959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:16.686961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:16.687050 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:16.687471 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:16.687474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:16.687578 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 21:21:16.687683 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:21:16.687823 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376}
I20260928 21:21:16.688805 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.678632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.5.18.253:40807)
0928 21:21:16.678747 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.678752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.678822 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:21:16.679003 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.679006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.679055 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.679108 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.679991 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.680694 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.681784 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.682122 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.682193 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.685312 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.685317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.685320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.685347 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.686938 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.687374 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.687379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.687380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.687469 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.687760 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.687763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.687765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.687975 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.688172 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.688520 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.688645 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:21:16.689994 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: No bootstrap required, opened a new log
I20260928 21:21:16.691516 27679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: Bootstrap starting.
I20260928 21:21:16.692219 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:16.693845 27679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:16.693854 27604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.694078 27604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea0a47596624614bb752ef6f266be94, State: Initialized, Role: FOLLOWER
I20260928 21:21:16.694383 27604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.694744 27679 log.cc:824] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:16.693552 27503 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a8de7da0d45a408aaeb3ba726b0546db"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42270
I20260928 21:21:16.694883 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.695175 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.695679 27604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:16.696712 27679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: No bootstrap required, opened a new log
I20260928 21:21:16.699388 27529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: No bootstrap required, opened a new log
I20260928 21:21:16.700557 27679 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.700805 27679 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc0ffa635674501a057f32af8c5fa5f, State: Initialized, Role: FOLLOWER
I20260928 21:21:16.701128 27679 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.702522 27679 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:16.702452 27529 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.702668 27529 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8de7da0d45a408aaeb3ba726b0546db, State: Initialized, Role: FOLLOWER
W20260928 21:21:16.702963 27699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:16.703081 27699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:16.699621 27536 master_runner.cc:186] Error getting master registration for 127.5.18.254:38607: OK, instance_id { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" instance_seqno: 1790630476206166 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:21:16.703258 27529 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.703461 27536 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:21:16.704650 27529 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:16.704741 27503 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:21:16.704859 27503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:16.704991 27701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.705271 27701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.706175 27503 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.707104 27503 leader_election.cc:288] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [CANDIDATE]: Term 1 election: Requested vote from peers 1ea0a47596624614bb752ef6f266be94 (127.5.18.253:40807), bcc0ffa635674501a057f32af8c5fa5f (127.5.18.252:33511)
W20260928 21:21:16.708516 27713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:16.708599 27713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:16.708827 27578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8de7da0d45a408aaeb3ba726b0546db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ea0a47596624614bb752ef6f266be94"
I20260928 21:21:16.709054 27578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:16.710100 27653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8de7da0d45a408aaeb3ba726b0546db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bcc0ffa635674501a057f32af8c5fa5f"
I20260928 21:21:16.710289 27653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:16.711534 27653 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8de7da0d45a408aaeb3ba726b0546db in term 1.
I20260928 21:21:16.712531 27480 leader_election.cc:302] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8de7da0d45a408aaeb3ba726b0546db, bcc0ffa635674501a057f32af8c5fa5f; no voters:
W20260928 21:21:16.713013 27724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:16.713092 27724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:16.713341 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.713466 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.713546 27705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:16.713644 27705 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [term 1 LEADER]: Becoming Leader. State: Replica: a8de7da0d45a408aaeb3ba726b0546db, State: Running, Role: LEADER
I20260928 21:21:16.713866 27578 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8de7da0d45a408aaeb3ba726b0546db in term 1.
I20260928 21:21:16.713963 27705 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } }
I20260928 21:21:16.716213 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8de7da0d45a408aaeb3ba726b0546db. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.716331 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:16.716681 27733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:16.718868 27611 master_runner.cc:263] No leader master found from master bcc0ffa635674501a057f32af8c5fa5f
I20260928 21:21:16.718935 27611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:21:16.720063 27461 master_runner.cc:263] No leader master found from master a8de7da0d45a408aaeb3ba726b0546db
I20260928 21:21:16.720137 27461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:21:16.721100 27733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:16.727124 27653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [term 1 FOLLOWER]: Refusing update from remote peer a8de7da0d45a408aaeb3ba726b0546db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:21:16.728157 27731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:21:16.733165 27701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8de7da0d45a408aaeb3ba726b0546db. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.733309 27701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.733810 27578 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [term 1 FOLLOWER]: Refusing update from remote peer a8de7da0d45a408aaeb3ba726b0546db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:21:16.734519 27731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:21:16.738533 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8de7da0d45a408aaeb3ba726b0546db. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.738672 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.742074 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.719702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.5.18.252:33511)
0928 21:21:16.719809 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.719814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.725967 (+ 6153us) server_negotiation.cc:407] Connection header received
0928 21:21:16.727640 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.727643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.727693 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.727767 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.734293 (+ 6526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.735097 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.736358 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.736693 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.736763 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.738564 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.738568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.738571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.738597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.740249 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.740746 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.740749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.740751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.740836 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.741112 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.741115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.741117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.741334 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.741453 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.741801 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.741922 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:21:16.743161 27731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.743285 27731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:16.744359 27701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.744474 27701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.745003 27737 mvcc.cc:205] Tried to move back new op lower bound from 7334422432665341952 to 7334422432625713152. Current Snapshot: MvccSnapshot[applied={T|T < 7334422432665341952}]
I20260928 21:21:16.745152 27738 mvcc.cc:205] Tried to move back new op lower bound from 7334422432665341952 to 7334422432625713152. Current Snapshot: MvccSnapshot[applied={T|T < 7334422432665341952}]
I20260928 21:21:16.745842 27733 catalog_manager.cc:1347] Generated new cluster ID: e517a46a6f8b4b258bf95849f68db582
I20260928 21:21:16.745899 27733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:16.752683 27531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.719499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.5.18.254:38607)
0928 21:21:16.719628 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.719633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.726343 (+ 6710us) server_negotiation.cc:407] Connection header received
0928 21:21:16.726466 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.726469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.726520 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.726597 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.742373 (+ 15776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.743140 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.744412 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.744757 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.744840 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.746843 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.746848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.746851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.746881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.748583 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.751171 (+ 2588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.751175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.751177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.751295 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.751627 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.751630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.751632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.751862 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.751986 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.752363 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.752500 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:21:16.760111 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:16.720727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.5.18.253:40807)
0928 21:21:16.720834 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.720839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.729868 (+ 9029us) server_negotiation.cc:407] Connection header received
0928 21:21:16.730062 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.730065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.730120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.730174 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.738861 (+ 8687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.739631 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.753677 (+ 14046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:16.754029 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:16.754120 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:16.756049 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:16.756053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:16.756056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:16.756088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:16.757794 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.758308 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.758312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.758314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.758404 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:16.758706 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:16.758709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:16.758711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:16.758936 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:16.759071 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:16.759812 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 21:21:16.759930 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:21:16.763373 27731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.763537 27731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:16.764859 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a8de7da0d45a408aaeb3ba726b0546db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8de7da0d45a408aaeb3ba726b0546db" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 38607 } } peers { permanent_uuid: "1ea0a47596624614bb752ef6f266be94" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 40807 } } peers { permanent_uuid: "bcc0ffa635674501a057f32af8c5fa5f" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 33511 } } }
I20260928 21:21:16.765015 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:16.767217 27733 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:16.768468 27733 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:16.771727 27740 mvcc.cc:205] Tried to move back new op lower bound from 7334422432665341952 to 7334422432625713152. Current Snapshot: MvccSnapshot[applied={T|T < 7334422432665341952 or (T in {7334422432665341952})}]
I20260928 21:21:16.784041 27733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8de7da0d45a408aaeb3ba726b0546db: Generated new TSK 0
I20260928 21:21:16.785008 27733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:21:16.832002 27531 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.828489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.5.18.254:38607)
0928 21:21:16.828629 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.828635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.830794 (+ 2159us) server_negotiation.cc:407] Connection header received
0928 21:21:16.831022 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.831026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.831081 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.831138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.831844 (+ 706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42292: BlockingRecv error: recv got EOF from 127.0.0.1:42292 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 21:21:16.833431 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.829060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.5.18.252:33511)
0928 21:21:16.829181 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.829186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.832429 (+ 3243us) server_negotiation.cc:407] Connection header received
0928 21:21:16.832612 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.832615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.832667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.832722 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.833302 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52564: BlockingRecv error: recv got EOF from 127.0.0.1:52564 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:21:16.835110 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.829904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.5.18.253:40807)
0928 21:21:16.830020 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.830026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.833980 (+ 3954us) server_negotiation.cc:407] Connection header received
0928 21:21:16.834321 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.834325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.834382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.834438 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.834988 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 21:21:16.840054 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.838429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.5.18.253:40807)
0928 21:21:16.838539 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.838544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.839016 (+ 472us) server_negotiation.cc:407] Connection header received
0928 21:21:16.839262 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.839266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.839348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.839407 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.839939 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54006: BlockingRecv error: recv got EOF from 127.0.0.1:54006 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:21:16.845794 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.843931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.5.18.253:40807)
0928 21:21:16.844054 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.844060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.844670 (+ 610us) server_negotiation.cc:407] Connection header received
0928 21:21:16.844893 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.844897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.844956 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.845013 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.845657 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54012: BlockingRecv error: recv got EOF from 127.0.0.1:54012 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:21:16.852218 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.850373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.5.18.253:40807)
0928 21:21:16.850487 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.850493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.851103 (+ 610us) server_negotiation.cc:407] Connection header received
0928 21:21:16.851321 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.851325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.851382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.851439 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.852085 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54026: BlockingRecv error: recv got EOF from 127.0.0.1:54026 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:21:16.860478 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.858683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.5.18.253:40807)
0928 21:21:16.858834 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.858839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.859414 (+ 575us) server_negotiation.cc:407] Connection header received
0928 21:21:16.859632 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.859635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.859693 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.859753 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.860354 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54036: BlockingRecv error: recv got EOF from 127.0.0.1:54036 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 21:21:16.873461 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.871422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.5.18.253:40807)
0928 21:21:16.871559 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.871565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.872269 (+ 704us) server_negotiation.cc:407] Connection header received
0928 21:21:16.872550 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.872555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.872615 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.872671 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.873302 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54052: BlockingRecv error: recv got EOF from 127.0.0.1:54052 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 21:21:16.894599 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.892569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.5.18.253:40807)
0928 21:21:16.892710 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.892716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.893373 (+ 657us) server_negotiation.cc:407] Connection header received
0928 21:21:16.893640 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.893645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.893707 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.893770 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.894445 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54066: BlockingRecv error: recv got EOF from 127.0.0.1:54066 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 21:21:16.930590 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.929322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.5.18.253:40807)
0928 21:21:16.929463 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.929469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.929644 (+ 175us) server_negotiation.cc:407] Connection header received
0928 21:21:16.929816 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.929820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.929879 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.929957 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.930459 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54078: BlockingRecv error: recv got EOF from 127.0.0.1:54078 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 21:21:16.997761 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:16.996542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.5.18.253:40807)
0928 21:21:16.996664 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:16.996667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:16.996825 (+ 158us) server_negotiation.cc:407] Connection header received
0928 21:21:16.996965 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:16.996968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:16.997023 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:16.997122 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:16.997621 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54080: BlockingRecv error: recv got EOF from 127.0.0.1:54080 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 21:21:17.128928 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:17.127798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.5.18.253:40807)
0928 21:21:17.127962 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.127967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.128073 (+ 106us) server_negotiation.cc:407] Connection header received
0928 21:21:17.128236 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.128239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.128286 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.128368 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:17.128828 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54092: BlockingRecv error: recv got EOF from 127.0.0.1:54092 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 21:21:17.388275 27609 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:17.387152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.5.18.253:40807)
0928 21:21:17.387310 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.387315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.387332 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:21:17.387530 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.387533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.387585 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.387665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:17.388145 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54096: BlockingRecv error: recv got EOF from 127.0.0.1:54096 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:21:17.704473 27699 catalog_manager.cc:1259] Loaded cluster ID: e517a46a6f8b4b258bf95849f68db582
I20260928 21:21:17.704555 27699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: loading cluster ID for follower catalog manager: success
I20260928 21:21:17.706275 27699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: acquiring CA information for follower catalog manager: success
I20260928 21:21:17.707435 27699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ea0a47596624614bb752ef6f266be94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:21:17.709717 27713 catalog_manager.cc:1259] Loaded cluster ID: e517a46a6f8b4b258bf95849f68db582
I20260928 21:21:17.709781 27713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: loading cluster ID for follower catalog manager: success
I20260928 21:21:17.711337 27713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: acquiring CA information for follower catalog manager: success
I20260928 21:21:17.712407 27713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcc0ffa635674501a057f32af8c5fa5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:21:17.906687 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:17.902581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.5.18.253:40807)
0928 21:21:17.902797 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.902802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.902858 (+ 56us) server_negotiation.cc:407] Connection header received
0928 21:21:17.902991 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.902993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.903130 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.903242 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:17.903641 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.904311 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.905216 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.905451 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.905529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:17.905810 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:17.906364 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:17.906453 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:17.906505 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:21:17.906557 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260928 21:21:17.912770 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:17.909525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.5.18.252:33511)
0928 21:21:17.909735 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.909738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.909751 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:21:17.909846 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.909848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.909954 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.910045 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:17.910196 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.910687 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.911479 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.911690 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.911766 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:17.912017 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:17.912430 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:17.912520 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:17.912604 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 21:21:17.912650 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 21:21:17.919415 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:17.915583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.5.18.252:33511)
0928 21:21:17.915681 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.915685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.916001 (+ 316us) server_negotiation.cc:407] Connection header received
0928 21:21:17.916104 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.916107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.916153 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.916219 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:17.916452 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.917206 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.918398 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.918643 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.918739 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:17.918988 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:17.919067 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:17.919132 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:17.919226 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:21:17.919282 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:21:17.919464 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:17.915805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.5.18.253:40807)
0928 21:21:17.915915 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.915920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.916021 (+ 101us) server_negotiation.cc:407] Connection header received
0928 21:21:17.916168 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.916171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.916219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.916281 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:17.916616 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.917355 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.918485 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.918706 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.918791 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:17.919027 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:17.919099 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:17.919157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:17.919308 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:21:17.919361 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:21:17.919865 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:17.915512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.5.18.254:38607)
0928 21:21:17.915768 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:17.915773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:17.916017 (+ 244us) server_negotiation.cc:407] Connection header received
0928 21:21:17.916112 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:17.916114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:17.916266 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:17.916371 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:17.916587 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.917098 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.917952 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:17.918178 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:17.918260 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:17.918939 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:17.919272 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:17.919361 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:17.919658 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 21:21:17.919731 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
I20260928 21:21:17.923084 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27461
I20260928 21:21:17.928380 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27536
I20260928 21:21:17.932976 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27611
2026-09-28T21:21:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2972 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:21:17 dist-test-slave-rc9f krb5kdc[27828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:21:17 dist-test-slave-rc9f krb5kdc[27828](info): set up 2 sockets
Sep 28 21:21:17 dist-test-slave-rc9f krb5kdc[27828](info): commencing operation
krb5kdc: starting...
W20260928 21:21:18.881795 5195 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/test-user.keytab.
Sep 28 21:21:18 dist-test-slave-rc9f krb5kdc[27828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:21:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.18.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
Entry for principal kudu/127.5.18.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab.
I20260928 21:21:19.000488 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46281
--webserver_interface=127.5.18.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:21:19.106906 27841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:19.107215 27841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:19.107250 27841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:19.107312 27841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:19.111259 27841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:19.111312 27841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:19.111344 27841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:19.111366 27841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:19.111394 27841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:19.116458 27841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.254.keytab
--principal=kudu/127.5.18.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.254:46281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.18.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:19.117445 27841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:19.118459 27841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:19.126754 27841 server_base.cc:1034] running on GCE node
W20260928 21:21:19.126722 27847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.126714 27849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.126739 27846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:19.127656 27841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:19.128747 27841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:19.129899 27841 hybrid_clock.cc:648] HybridClock initialized: now 1790630479129880 us; error 37 us; skew 500 ppm
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:19.132624 27841 init.cc:377] Logged in from keytab as kudu/127.5.18.254@KRBTEST.COM (short username kudu)
I20260928 21:21:19.133584 27841 webserver.cc:460] Webserver started at http://127.5.18.254:32913/ using document root <none> and password file <none>
I20260928 21:21:19.134136 27841 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:19.134200 27841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:19.134388 27841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:19.136433 27841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data/instance:
uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "bd30e57fa61d4bc4149937cc1c304738"
server_key_iv: "95488cac1700a900e81a7cee3e3d748b"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.136968 27841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance:
uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "bd30e57fa61d4bc4149937cc1c304738"
server_key_iv: "95488cac1700a900e81a7cee3e3d748b"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.140859 27841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:21:19.143162 27856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:19.144335 27841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:19.144457 27841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal
uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "bd30e57fa61d4bc4149937cc1c304738"
server_key_iv: "95488cac1700a900e81a7cee3e3d748b"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.144549 27841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:19.160446 27841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:19.163520 27841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:19.163691 27841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:19.170116 27841 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.254:46281
I20260928 21:21:19.170359 27908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.254:46281 every 8 connection(s)
I20260928 21:21:19.171017 27841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/data/info.pb
I20260928 21:21:19.173637 27909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:19.176237 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27841
I20260928 21:21:19.176378 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-0/wal/instance
I20260928 21:21:19.176671 5195 external_mini_cluster.cc:1428] Setting key 971acf558c3761ee3eb31de6361a6d12
I20260928 21:21:19.177759 27909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.18.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.253@KRBTEST.COM" created.
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:19.192404 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.180236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46281 (local address 127.0.0.1:52446)
0928 21:21:19.180739 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.180781 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.181648 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.182274 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.182279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.182685 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.182913 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.182924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.183999 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.184002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.184716 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.184724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.185328 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.185332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.185423 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.186006 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.186027 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.187613 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.189525 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.189533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.189541 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.189781 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.190101 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.190102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.190104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.190208 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.190945 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.191054 (+ 109us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.191202 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.191340 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.191618 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260928 21:21:19.192408 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.181195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.5.18.254:46281)
0928 21:21:19.181483 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.181494 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.181535 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:21:19.181916 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.181919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.182107 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.182263 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.183047 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.183887 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.184820 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.185197 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.185333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.187762 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.187768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.187778 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.187799 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.189425 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.189879 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.189885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.189887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.190022 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.190308 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.190311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.190313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.190843 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.191005 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.191477 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.191721 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
I20260928 21:21:19.194967 27909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:19.195667 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.195430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40942)
0928 21:21:19.195536 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.195605 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 21:21:19.195824 27860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
I20260928 21:21:19.196605 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.196395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40950)
0928 21:21:19.196511 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.196553 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 21:21:19.196723 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:45785: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
Entry for principal kudu/127.5.18.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab.
I20260928 21:21:19.203850 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:45785
--webserver_interface=127.5.18.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:21:19.245976 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } attempt: 1
I20260928 21:21:19.246938 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.246583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40958)
0928 21:21:19.246784 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.246852 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 21:21:19.247836 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.247557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40964)
0928 21:21:19.247680 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.247744 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:21:19.248018 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:45785: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
W20260928 21:21:19.313833 27916 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:19.314122 27916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:19.314143 27916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:19.314184 27916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:21:19.316206 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } attempt: 2
I20260928 21:21:19.317060 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.316735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40968)
0928 21:21:19.316897 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.316967 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 21:21:19.317806 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.317602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40978)
0928 21:21:19.317694 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.317743 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:21:19.317955 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.253:45785: Network error: Client connection negotiation failed: client connection to 127.5.18.253:45785: connect: Connection refused (error 111)
W20260928 21:21:19.318598 27916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:19.318653 27916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:19.318679 27916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:19.318709 27916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:19.318737 27916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:19.324110 27916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.253.keytab
--principal=kudu/127.5.18.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.253:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.18.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:19.325166 27916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:19.326225 27916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:21:19.334492 27922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.334486 27924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.334484 27921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:19.334775 27916 server_base.cc:1034] running on GCE node
I20260928 21:21:19.335371 27916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:19.336444 27916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:19.337610 27916 hybrid_clock.cc:648] HybridClock initialized: now 1790630479337592 us; error 29 us; skew 500 ppm
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:19.340308 27916 init.cc:377] Logged in from keytab as kudu/127.5.18.253@KRBTEST.COM (short username kudu)
I20260928 21:21:19.341257 27916 webserver.cc:460] Webserver started at http://127.5.18.253:36729/ using document root <none> and password file <none>
I20260928 21:21:19.341853 27916 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:19.341920 27916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:19.342090 27916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:19.344120 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data/instance:
uuid: "9e24e93fa300466c881c01ac9f0d81ed"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "095fb55ef6a3ccdc900dc27c3231d7ee"
server_key_iv: "8634dc67d1561f01fd01681bfe3fbaf0"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.344662 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal/instance:
uuid: "9e24e93fa300466c881c01ac9f0d81ed"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "095fb55ef6a3ccdc900dc27c3231d7ee"
server_key_iv: "8634dc67d1561f01fd01681bfe3fbaf0"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.348474 27916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:21:19.350860 27931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:19.352056 27916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:19.352164 27916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal
uuid: "9e24e93fa300466c881c01ac9f0d81ed"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "095fb55ef6a3ccdc900dc27c3231d7ee"
server_key_iv: "8634dc67d1561f01fd01681bfe3fbaf0"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.352284 27916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:19.374671 27916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:19.377841 27916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:19.378017 27916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:19.384603 27916 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.253:45785
I20260928 21:21:19.384860 27983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.253:45785 every 8 connection(s)
I20260928 21:21:19.385543 27916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/data/info.pb
I20260928 21:21:19.388259 27984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:19.389876 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27916
I20260928 21:21:19.390003 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-1/wal/instance
I20260928 21:21:19.390304 5195 external_mini_cluster.cc:1428] Setting key 23759f74dc89e6f6ba27e856181bfdc4
I20260928 21:21:19.392601 27984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.18.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.18.252@KRBTEST.COM" created.
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:19.406132 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.394642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.5.18.254:46281)
0928 21:21:19.394763 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.394766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.395823 (+ 1057us) server_negotiation.cc:407] Connection header received
0928 21:21:19.396733 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.396736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.396797 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.396866 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.398121 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.398649 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.399879 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.400110 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.400209 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.402716 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.402720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.402723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.402753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.403989 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.404577 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.404580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.404581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.404647 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.405000 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.405002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.405003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.405162 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.405230 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.405905 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.406002 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:21:19.407259 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.395184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46281 (local address 127.0.0.1:52454)
0928 21:21:19.395679 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.395720 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.396540 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.397055 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.397062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.397389 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.397983 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.398000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.398791 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.398795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.399773 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.399782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.400248 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.400252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.400442 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.401009 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.401029 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.402551 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.404105 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.404112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.404128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.404485 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.404729 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.404732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.404734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.404918 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.405253 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.405260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.405642 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.405861 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.406198 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":152,"threads_started":1}
I20260928 21:21:19.408615 27984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:19.409215 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } attempt: 3
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
Entry for principal kudu/127.5.18.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab.
I20260928 21:21:19.416791 5195 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
/tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:41815
--webserver_interface=127.5.18.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.18.212:40615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:21:19.417819 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.409117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40988)
0928 21:21:19.409225 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.409240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.409325 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.409895 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.409897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.410019 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.410180 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.410188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.411204 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.411207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.411791 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.411799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.412375 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.412378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.412471 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.412896 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.412910 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.414233 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.416192 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.416195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.416197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.416427 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.416708 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.416710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.416712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.416786 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.417256 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.417259 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.417366 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.417473 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.417638 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":21}
I20260928 21:21:19.417891 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.409163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.5.18.253:45785)
0928 21:21:19.409481 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.409492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.409513 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:21:19.409552 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.409557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.409772 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.409903 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.410301 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.411107 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.411883 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.412284 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.412370 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.414346 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.414355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.414368 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.414391 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.416088 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.416532 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.416537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.416539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.416624 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.416866 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.416869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.416872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.417169 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.417299 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.417560 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.417727 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260928 21:21:19.418779 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.409525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:40996)
0928 21:21:19.409654 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.409667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.409743 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.410081 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.410084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.410235 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.410333 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.410339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.411248 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.411252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.411758 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.411765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.412408 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.412412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.412514 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.413043 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.413054 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.414701 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.417209 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.417213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.417215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.417458 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.417764 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.417768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.417770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.417855 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.418279 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.418282 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.418388 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.418495 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.418625 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:21:19.418817 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.409556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.5.18.253:45785)
0928 21:21:19.409876 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.409884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.409895 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:21:19.409921 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.409933 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.409974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.410044 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.410425 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.411146 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.411868 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.412307 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.412398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.414874 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.414878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.414880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.414904 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.417109 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.417553 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.417556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.417558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.417663 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.417969 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.417972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.417980 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.418153 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.418284 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.418561 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.418660 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":232,"spinlock_wait_cycles":1664,"thread_start_us":143,"threads_started":1}
I20260928 21:21:19.420387 27909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:19.420492 27984 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:19.421082 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.420865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42468)
0928 21:21:19.420981 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.421028 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:21:19.421645 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.421121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42476)
0928 21:21:19.421487 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.421582 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260928 21:21:19.421834 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.421622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42480)
0928 21:21:19.421738 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.421783 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 21:21:19.421898 27934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
W20260928 21:21:19.422092 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
I20260928 21:21:19.422721 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.422515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42484)
0928 21:21:19.422613 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.422662 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 21:21:19.422863 27984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
I20260928 21:21:19.453244 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 1
I20260928 21:21:19.454074 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.453794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42490)
0928 21:21:19.453934 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.453994 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 21:21:19.454735 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.454557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42504)
0928 21:21:19.454641 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.454682 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 21:21:19.454871 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
I20260928 21:21:19.472098 27984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 1
I20260928 21:21:19.472961 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.472634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42510)
0928 21:21:19.472795 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.472868 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 21:21:19.473652 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.473418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42516)
0928 21:21:19.473516 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.473591 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 21:21:19.473774 27984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
W20260928 21:21:19.525272 27991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:21:19.525543 27991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:21:19.525588 27991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:21:19.525640 27991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:21:19.529632 27991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:21:19.529701 27991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:21:19.529719 27991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:21:19.529733 27991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:21:19.529747 27991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:21:19.530055 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 2
I20260928 21:21:19.530856 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.530572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42522)
0928 21:21:19.530713 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.530776 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:21:19.531548 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.531350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42536)
0928 21:21:19.531447 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.531492 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 21:21:19.531677 27909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
I20260928 21:21:19.535375 27991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.18.212:40615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.18.254:46281,127.5.18.253:45785,127.5.18.252:41815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/krb5kdc/kudu_127.5.18.252.keytab
--principal=kudu/127.5.18.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.18.252:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.18.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:21:19.536379 27991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:21:19.537366 27991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:21:19.541913 27984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 2
I20260928 21:21:19.542646 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.542399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42542)
0928 21:21:19.542523 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.542579 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:21:19.543226 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.543056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42546)
0928 21:21:19.543134 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.543175 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:21:19.543324 27984 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.18.252:41815: Network error: Client connection negotiation failed: client connection to 127.5.18.252:41815: connect: Connection refused (error 111)
W20260928 21:21:19.545672 28000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.545672 27997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:21:19.545668 27998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:21:19.546109 27991 server_base.cc:1034] running on GCE node
I20260928 21:21:19.546682 27991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:21:19.547887 27991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:21:19.549062 27991 hybrid_clock.cc:648] HybridClock initialized: now 1790630479549044 us; error 28 us; skew 500 ppm
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:21:19.551883 27991 init.cc:377] Logged in from keytab as kudu/127.5.18.252@KRBTEST.COM (short username kudu)
I20260928 21:21:19.552932 27991 webserver.cc:460] Webserver started at http://127.5.18.252:34911/ using document root <none> and password file <none>
I20260928 21:21:19.553468 27991 fs_manager.cc:359] Metadata directory not provided
I20260928 21:21:19.553539 27991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:21:19.553776 27991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:21:19.555903 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data/instance:
uuid: "ff9e5fa2125f480bbab1edf4967006b5"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "99b064266d4aba97a082ebaa8423cab1"
server_key_iv: "5ecb731f1cb6fdc06aece02a8736c931"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.556473 27991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal/instance:
uuid: "ff9e5fa2125f480bbab1edf4967006b5"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "99b064266d4aba97a082ebaa8423cab1"
server_key_iv: "5ecb731f1cb6fdc06aece02a8736c931"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.560353 27991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:21:19.562762 28007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:21:19.563995 27991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:21:19.564117 27991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data,/tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal
uuid: "ff9e5fa2125f480bbab1edf4967006b5"
format_stamp: "Formatted at 2026-09-28 21:21:19 on dist-test-slave-rc9f"
server_key: "99b064266d4aba97a082ebaa8423cab1"
server_key_iv: "5ecb731f1cb6fdc06aece02a8736c931"
server_key_version: "encryptionkey@0"
I20260928 21:21:19.564214 27991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:21:19.576763 27991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:21:19.579577 27991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:21:19.579758 27991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:21:19.586443 27991 rpc_server.cc:307] RPC server started. Bound to: 127.5.18.252:41815
I20260928 21:21:19.586690 28059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.18.252:41815 every 8 connection(s)
I20260928 21:21:19.587360 27991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/data/info.pb
I20260928 21:21:19.589973 28060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:21:19.593047 5195 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu as pid 27991
I20260928 21:21:19.593173 5195 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyuKJ5W/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790630206151656-5195-0/minicluster-data/master-2/wal/instance
I20260928 21:21:19.593442 5195 external_mini_cluster.cc:1428] Setting key b39a4e0c476090bd8aa8c180ae09e09b
I20260928 21:21:19.593644 28060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:21:19.605597 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.595864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.5.18.254:46281)
0928 21:21:19.595991 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.595996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.596011 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:21:19.596230 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.596233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.596285 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.596350 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.597252 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.597818 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.598826 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.599095 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.599170 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.601826 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.601830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.601834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.601862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.603325 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.604011 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.604014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.604016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.604091 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.604455 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.604458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.604460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.604675 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.604773 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.605284 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.605407 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:21:19.606451 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.595290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.254:46281 (local address 127.0.0.1:52470)
0928 21:21:19.595667 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.595717 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.596047 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.596399 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.596408 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.596839 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.597112 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.597130 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.597911 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.597914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.598722 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.598730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.599235 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.599239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.599408 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.600037 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.600052 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.601684 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.603428 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.603435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.603445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.603877 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.604186 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.604190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.604192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.604362 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.604774 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.604780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.605043 (+ 263us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.605202 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.605543 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":12,"spinlock_wait_cycles":5632,"thread_start_us":132,"threads_started":1}
I20260928 21:21:19.606449 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.594768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.5.18.252:41815)
0928 21:21:19.595169 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.595183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.595216 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:21:19.596047 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.596068 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.596300 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.596464 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.597162 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.598163 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.599020 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.599544 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.599649 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.602280 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.602286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.602295 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.602315 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.603885 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.604260 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.604265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.604266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.604360 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.604600 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.604603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.604604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.605041 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.605180 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.605421 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.605614 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":271,"spinlock_wait_cycles":2432,"thread_start_us":106,"threads_started":1}
I20260928 21:21:19.607550 28060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.254@KRBTEST.COM
I20260928 21:21:19.615041 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.607991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.253:45785 (local address 127.0.0.1:41000)
0928 21:21:19.608112 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.608126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.608190 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.608514 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.608517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.608676 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.608780 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.608786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.609515 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.609518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.610077 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.610085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.610526 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.610529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.610620 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.611034 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.611044 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.612150 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.613738 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.613741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.613743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.613959 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.614199 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.614202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.614204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.614288 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.614609 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.614612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.614705 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.614792 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.614890 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:21:19.615121 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.608181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41000 (local address 127.5.18.253:45785)
0928 21:21:19.608302 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.608306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.608319 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:21:19.608361 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.608364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.608409 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.608477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.608894 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.609419 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.610184 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.610432 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.610520 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.612280 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.612282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.612284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.612307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.613644 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.614045 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.614047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.614049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.614122 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.614361 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.614363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.614364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.614534 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.614601 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.614871 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.614993 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:21:19.615708 28060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:21:19.616312 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.608603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.5.18.254:46281)
0928 21:21:19.608695 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.608699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.608733 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:21:19.608850 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.608853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.608891 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.608948 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.609481 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.610004 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.610709 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.611007 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.611096 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.613147 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.613151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.613154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.613177 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.614589 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.614972 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.614976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.614978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.615053 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.615537 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.615540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.615542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.615745 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.615959 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.616075 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.616177 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.252@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.18.253@KRBTEST.COM
I20260928 21:21:19.623183 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.616194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.5.18.252:41815)
0928 21:21:19.616279 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.616283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.616551 (+ 268us) server_negotiation.cc:407] Connection header received
0928 21:21:19.616635 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.616637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.616673 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.616731 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.617103 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.617712 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.618392 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.618628 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.618708 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.620371 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.620375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.620377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.620397 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.621769 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.622120 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.622124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.622125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.622194 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.622449 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.622452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.622454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.622636 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.622708 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.622968 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.623057 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":30}
I20260928 21:21:19.623185 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.616164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42564)
0928 21:21:19.616495 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.616508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.616573 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.616766 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.616768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.616915 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.617001 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.617007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.617797 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.617800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.618298 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.618305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.618713 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.618716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.618829 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.619216 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.619225 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.620260 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.621857 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.621859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.621860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.622030 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.622283 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.622286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.622288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.622369 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.622719 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.622722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.622810 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.622906 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.623012 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":20}
I20260928 21:21:19.625087 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.617401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.5.18.253:45785)
0928 21:21:19.617499 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.617503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.617515 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:21:19.617551 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.617554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.617637 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.617699 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.618218 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.618681 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.619492 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.619703 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.619782 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.622143 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.622146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.622149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.622172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.623615 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.624034 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.624037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.624039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.624108 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.624361 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.624365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.624367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.624526 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.624713 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.624856 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.624955 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:21:19.626106 28060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: Bootstrap starting.
I20260928 21:21:19.628322 5195 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:21:19.628402 28060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:19.629072 28060 log.cc:824] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:19.629396 27883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52472
I20260928 21:21:19.630750 27909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 3
I20260928 21:21:19.631333 28060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: No bootstrap required, opened a new log
I20260928 21:21:19.634459 27984 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } attempt: 3
I20260928 21:21:19.634802 28060 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.635280 28060 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9e5fa2125f480bbab1edf4967006b5, State: Initialized, Role: FOLLOWER
I20260928 21:21:19.635614 28060 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.254@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:21:19.636186 28067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.636408 28067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.636919 28060 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 21:21:19 dist-test-slave-rc9f krb5kdc[27828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.18.253@KRBTEST.COM for kudu/127.5.18.252@KRBTEST.COM
I20260928 21:21:19.640228 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.631078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42572)
0928 21:21:19.631190 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.631204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.631286 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.631501 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.631504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.631685 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.631786 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.631792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.632701 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.632704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.633542 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.633549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.634078 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.634082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.634163 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.634741 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.634756 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.636156 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.638094 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.638097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.638111 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.638358 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.638954 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.638957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.638959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.639044 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.639740 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.639743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.639844 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.639934 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.640065 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 21:21:19.640759 28063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.631128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.5.18.252:41815)
0928 21:21:19.631243 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.631247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.631262 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:21:19.631372 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.631375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.631417 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.631486 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.631891 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.632602 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.633648 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.633978 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.634079 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.636310 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.636314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.636317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.636341 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.637995 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.638487 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.638491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.638493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.638849 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.639127 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.639130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.639132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.639448 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.640441 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.640522 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.640628 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 21:21:19.643329 28078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:19.643409 28078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:19.640625 27991 master_runner.cc:186] Error getting master registration for 127.5.18.254:46281: OK, instance_id { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" instance_seqno: 1790630479168374 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:21:19.644331 27991 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:21:19.644654 27909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: Bootstrap starting.
I20260928 21:21:19.647501 27909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:19.648437 27909 log.cc:824] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:19.650731 27909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: No bootstrap required, opened a new log
I20260928 21:21:19.650867 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.634885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.18.252:41815 (local address 127.0.0.1:42576)
0928 21:21:19.635012 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:21:19.635027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:21:19.635100 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:21:19.635456 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:21:19.635460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:21:19.635627 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:21:19.635727 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.635733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.636878 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.636882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.637704 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:21:19.637717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.638272 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.638276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:21:19.638360 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.638793 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:21:19.638808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:21:19.640157 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:21:19.648903 (+ 8746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.648908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.648911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.649242 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.649717 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:21:19.649721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:21:19.649724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:21:19.649837 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:21:19.650290 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:21:19.650293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:21:19.650418 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:21:19.650532 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:21:19.650695 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:21:19.651548 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.634914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.5.18.252:41815)
0928 21:21:19.635192 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.635200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.635214 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:21:19.635245 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.635248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.635286 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.635420 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.635903 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.636747 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.637842 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.638173 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.638253 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.640330 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.640334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.640336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.640358 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.642029 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.649396 (+ 7367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.649401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.649403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.649518 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.649935 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.649939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.649940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.650175 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.650286 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.651226 (+ 940us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.651379 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
I20260928 21:21:19.653832 27909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.654052 27909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517ff3b8d47945d7ae4baa8e6dd1a95c, State: Initialized, Role: FOLLOWER
I20260928 21:21:19.654379 27909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.654902 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.655100 27984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: Bootstrap starting.
I20260928 21:21:19.655124 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.655719 27909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:19.655802 27883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:21:19.655886 27883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:19.657104 27883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.657887 27984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: Neither blocks nor log segments found. Creating new log.
I20260928 21:21:19.658033 27883 leader_election.cc:288] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [CANDIDATE]: Term 1 election: Requested vote from peers 9e24e93fa300466c881c01ac9f0d81ed (127.5.18.253:45785), ff9e5fa2125f480bbab1edf4967006b5 (127.5.18.252:41815)
I20260928 21:21:19.658946 28034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ff9e5fa2125f480bbab1edf4967006b5"
I20260928 21:21:19.659183 28034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:19.660171 27984 log.cc:824] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: Log is configured to *not* fsync() on all Append() calls
I20260928 21:21:19.660485 28034 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517ff3b8d47945d7ae4baa8e6dd1a95c in term 1.
I20260928 21:21:19.662196 27859 leader_election.cc:302] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 517ff3b8d47945d7ae4baa8e6dd1a95c, ff9e5fa2125f480bbab1edf4967006b5; no voters:
I20260928 21:21:19.662441 27984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: No bootstrap required, opened a new log
I20260928 21:21:19.662782 28081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:21:19.662958 28081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [term 1 LEADER]: Becoming Leader. State: Replica: 517ff3b8d47945d7ae4baa8e6dd1a95c, State: Running, Role: LEADER
I20260928 21:21:19.663416 28081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
W20260928 21:21:19.665105 27862 tablet.cc:2369] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:21:19.665154 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517ff3b8d47945d7ae4baa8e6dd1a95c. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.665244 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:19.665611 28103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:21:19.667061 27984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.667294 27984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e24e93fa300466c881c01ac9f0d81ed, State: Initialized, Role: FOLLOWER
I20260928 21:21:19.667683 27984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } }
I20260928 21:21:19.663810 27841 master_runner.cc:186] Error getting master registration for 127.5.18.253:45785: OK, instance_id { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" instance_seqno: 1790630479382760 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:21:19.668473 27841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:21:19.669160 27984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:21:19.669471 27958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e24e93fa300466c881c01ac9f0d81ed"
I20260928 21:21:19.669716 27958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:21:19.669991 28103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:21:19.671051 27958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 517ff3b8d47945d7ae4baa8e6dd1a95c in term 1.
I20260928 21:21:19.671170 28105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.671303 28105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:21:19.674829 28116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:21:19.674963 28116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:21:19.685452 28066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.664010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.5.18.252:41815)
0928 21:21:19.664110 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.664114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.672669 (+ 8555us) server_negotiation.cc:407] Connection header received
0928 21:21:19.672935 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.672938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.672992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.673044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.676847 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.677619 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.678732 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.679082 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.679181 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.681157 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.681163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.681166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.681198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.683226 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.683812 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.683817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.683819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.683915 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.684345 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.684349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.684351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.684624 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.684751 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.685121 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.685275 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 21:21:19.685897 27913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.662049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.5.18.254:46281)
0928 21:21:19.662154 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.662158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.667387 (+ 5229us) server_negotiation.cc:407] Connection header received
0928 21:21:19.668324 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.668327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.668381 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.668431 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.677664 (+ 9233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.678354 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.679542 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.679854 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.679930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.681956 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.681961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.681964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.682003 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.683596 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.684174 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.684178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.684179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.684282 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.684591 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.684594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.684596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.684827 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.684947 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.685506 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.685746 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 21:21:19.694877 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:19.661105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.5.18.253:45785)
0928 21:21:19.661216 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.661221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.672455 (+ 11234us) server_negotiation.cc:407] Connection header received
0928 21:21:19.674039 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.674042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.674090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.674135 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.675020 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.675736 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.685682 (+ 9946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:19.686069 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:19.686161 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:19.691129 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:21:19.691134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:21:19.691137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:21:19.691171 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:21:19.692855 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.693398 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.693402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.693404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.693492 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:21:19.693823 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:21:19.693827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:21:19.693829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:21:19.694062 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:21:19.694183 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:19.694563 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 21:21:19.694696 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:21:19.698215 28034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [term 1 FOLLOWER]: Refusing update from remote peer 517ff3b8d47945d7ae4baa8e6dd1a95c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:21:19.699075 28100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:21:19.703897 28067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517ff3b8d47945d7ae4baa8e6dd1a95c. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.704056 28067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.707309 27958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [term 1 FOLLOWER]: Refusing update from remote peer 517ff3b8d47945d7ae4baa8e6dd1a95c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:21:19.707966 28100 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:21:19.712601 28100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.712749 28100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:19.714110 28105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517ff3b8d47945d7ae4baa8e6dd1a95c. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.714232 28105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.714356 28067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.714457 28067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.715098 28119 mvcc.cc:205] Tried to move back new op lower bound from 7334422444805632000 to 7334422444704841728. Current Snapshot: MvccSnapshot[applied={T|T < 7334422444805632000}]
I20260928 21:21:19.724198 28103 catalog_manager.cc:1347] Generated new cluster ID: b8de41064e2948f28f48436c0fecdd09
I20260928 21:21:19.724290 28103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:21:19.730211 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.730357 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c [sys.catalog]: This master's current role is: LEADER
I20260928 21:21:19.731587 28105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517ff3b8d47945d7ae4baa8e6dd1a95c" member_type: VOTER last_known_addr { host: "127.5.18.254" port: 46281 } } peers { permanent_uuid: "9e24e93fa300466c881c01ac9f0d81ed" member_type: VOTER last_known_addr { host: "127.5.18.253" port: 45785 } } peers { permanent_uuid: "ff9e5fa2125f480bbab1edf4967006b5" member_type: VOTER last_known_addr { host: "127.5.18.252" port: 41815 } } }
I20260928 21:21:19.731714 28105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:21:19.745121 28103 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:21:19.746551 28103 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:21:19.758320 28103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 517ff3b8d47945d7ae4baa8e6dd1a95c: Generated new TSK 0
I20260928 21:21:19.759284 28103 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:21:19.793488 28066 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.790133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.5.18.252:41815)
0928 21:21:19.790268 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.790273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.792355 (+ 2082us) server_negotiation.cc:407] Connection header received
0928 21:21:19.792569 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.792573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.792626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.792679 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.793333 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42594: BlockingRecv error: recv got EOF from 127.0.0.1:42594 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 21:21:19.795233 27913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.790612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.5.18.254:46281)
0928 21:21:19.790729 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.790733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.794280 (+ 3547us) server_negotiation.cc:407] Connection header received
0928 21:21:19.794485 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.794489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.794547 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.794600 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.795112 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52498: BlockingRecv error: recv got EOF from 127.0.0.1:52498 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:21:19.796739 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.791621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.5.18.253:45785)
0928 21:21:19.791732 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.791737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.795855 (+ 4118us) server_negotiation.cc:407] Connection header received
0928 21:21:19.796039 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.796042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.796100 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.796149 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.796625 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41034: BlockingRecv error: recv got EOF from 127.0.0.1:41034 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 21:21:19.802737 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.799740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.5.18.253:45785)
0928 21:21:19.799900 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.799904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.801820 (+ 1916us) server_negotiation.cc:407] Connection header received
0928 21:21:19.802026 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.802029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.802104 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.802153 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.802640 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41042: BlockingRecv error: recv got EOF from 127.0.0.1:41042 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 21:21:19.807818 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.806311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41058 (local address 127.5.18.253:45785)
0928 21:21:19.806411 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.806415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.806956 (+ 541us) server_negotiation.cc:407] Connection header received
0928 21:21:19.807133 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.807136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.807185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.807230 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.807715 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41058: BlockingRecv error: recv got EOF from 127.0.0.1:41058 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 21:21:19.813444 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.811957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41062 (local address 127.5.18.253:45785)
0928 21:21:19.812056 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.812060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.812565 (+ 505us) server_negotiation.cc:407] Connection header received
0928 21:21:19.812749 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.812752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.812802 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.812849 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.813325 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41062: BlockingRecv error: recv got EOF from 127.0.0.1:41062 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 21:21:19.821900 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.820408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.5.18.253:45785)
0928 21:21:19.820579 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.820584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.820880 (+ 296us) server_negotiation.cc:407] Connection header received
0928 21:21:19.821091 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.821094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.821150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.821202 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.821789 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41070: BlockingRecv error: recv got EOF from 127.0.0.1:41070 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 21:21:19.834489 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.832577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.5.18.253:45785)
0928 21:21:19.832696 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.832701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.833325 (+ 624us) server_negotiation.cc:407] Connection header received
0928 21:21:19.833552 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.833555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.833640 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.833694 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.834339 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41084: BlockingRecv error: recv got EOF from 127.0.0.1:41084 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 21:21:19.855912 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.854057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.5.18.253:45785)
0928 21:21:19.854179 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.854183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.854796 (+ 613us) server_negotiation.cc:407] Connection header received
0928 21:21:19.855021 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.855024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.855096 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.855145 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.855772 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41090: BlockingRecv error: recv got EOF from 127.0.0.1:41090 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:21:19.891469 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.890524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.5.18.253:45785)
0928 21:21:19.890654 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.890657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.890767 (+ 110us) server_negotiation.cc:407] Connection header received
0928 21:21:19.890883 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.890886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.890933 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.891014 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.891372 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41102: BlockingRecv error: recv got EOF from 127.0.0.1:41102 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:21:19.958575 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:19.957553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41114 (local address 127.5.18.253:45785)
0928 21:21:19.957735 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:19.957739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:19.957818 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:21:19.957945 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:19.957955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:19.958014 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:19.958085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:19.958453 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41114: BlockingRecv error: recv got EOF from 127.0.0.1:41114 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 21:21:20.089588 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:20.088462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.5.18.253:45785)
0928 21:21:20.088616 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.088620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.088734 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:21:20.088879 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.088883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.088948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.089015 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:20.089448 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41124: BlockingRecv error: recv got EOF from 127.0.0.1:41124 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 21:21:20.348666 27990 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:21:20.347531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.5.18.253:45785)
0928 21:21:20.347672 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.347677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.347754 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:21:20.347921 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.347925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.347973 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.348040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:21:20.348522 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41130: BlockingRecv error: recv got EOF from 127.0.0.1:41130 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:21:20.644726 28078 catalog_manager.cc:1259] Loaded cluster ID: b8de41064e2948f28f48436c0fecdd09
I20260928 21:21:20.644804 28078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: loading cluster ID for follower catalog manager: success
I20260928 21:21:20.646512 28078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: acquiring CA information for follower catalog manager: success
I20260928 21:21:20.647670 28078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff9e5fa2125f480bbab1edf4967006b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:21:20.676258 28116 catalog_manager.cc:1259] Loaded cluster ID: b8de41064e2948f28f48436c0fecdd09
I20260928 21:21:20.676344 28116 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: loading cluster ID for follower catalog manager: success
I20260928 21:21:20.678045 28116 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: acquiring CA information for follower catalog manager: success
I20260928 21:21:20.679180 28116 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e24e93fa300466c881c01ac9f0d81ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:21:20.866245 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:20.862480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.5.18.253:45785)
0928 21:21:20.862740 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.862744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.862759 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:21:20.862896 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.862898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.863020 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.863111 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:20.863364 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.863908 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.864786 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.865034 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.865124 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:20.865363 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:20.865865 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:20.865959 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:20.866037 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:21:20.866090 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260928 21:21:20.872615 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:20.869144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.5.18.252:41815)
0928 21:21:20.869372 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.869376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.869390 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:21:20.869497 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.869499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.869652 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.869756 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:20.869990 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.870502 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.871315 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.871529 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.871602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:20.871850 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:20.872257 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:20.872343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:20.872416 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:21:20.872463 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260928 21:21:20.878868 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:20.875380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.5.18.252:41815)
0928 21:21:20.875487 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.875491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.875678 (+ 187us) server_negotiation.cc:407] Connection header received
0928 21:21:20.875819 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.875828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.875878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.875937 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:20.876232 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.876850 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.877857 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.878068 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.878153 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:20.878416 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:20.878483 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:20.878539 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:20.878674 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 21:21:20.878724 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:21:20.879732 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:20.875960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.5.18.253:45785)
0928 21:21:20.876085 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.876089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.876274 (+ 185us) server_negotiation.cc:407] Connection header received
0928 21:21:20.876426 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.876430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.876480 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.876523 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:20.876870 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.877382 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.878634 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.878928 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.879028 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:20.879290 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:20.879366 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:20.879409 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:20.879554 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:21:20.879609 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:21:20.880645 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:21:20.875374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.5.18.254:46281)
0928 21:21:20.875606 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 21:21:20.875611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:21:20.875844 (+ 233us) server_negotiation.cc:407] Connection header received
0928 21:21:20.876052 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:21:20.876056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:21:20.876238 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:21:20.876372 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:21:20.876557 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.877337 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.879094 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:21:20.879355 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:21:20.879442 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:21:20.879894 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:21:20.880185 (+ 291us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:21:20.880252 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 21:21:20.880462 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 21:21:20.880521 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260928 21:21:20.887053 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27841
I20260928 21:21:20.892426 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27916
I20260928 21:21:20.897718 5195 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyuKJ5W/build/debug/bin/kudu with pid 27991
2026-09-28T21:21:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2964 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5937 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274739 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST